Define if post extract from a bilingual Facebook page are in English using Python I am currently extracting post on a biligual page on Facebook. Therefore, I have the problem of splitting the post in French and english before starting analysing them. I have construct a function which permit me to filter the sentence in english from all my sentence. 

However it is quite complex and not accurate..

### import enchant
import nltk
### nltk.download()
### from nltk.corpus import words
from nltk.corpus import brown
import pandas as pd
import numpy as np
import re
  ### pop is an example of posts...

pop = ['hi','say','bonjour mister','sauf','tu am      belle','I suis cool','Worst trip ever je deteste AirTransat! ce sont les pires']


### Defining my function

def Extract_sentence_english(list_of_sentence):

    ### Removing punctuation, numeric and whitespace at the same time
    list_of_sentence = [re.sub(r'[^A-Za-z]+', ' ', x) for x in list_of_sentence]
    ### To lower
    list_of_sentence = map(lambda x: x.lower(), list_of_sentence)

    ### Split the list into words (splitting on space)
    split_list = [y for x in list_of_sentence for y in x.split(' ')] 
    ### Return a logical value (True if English, False if other)
    log = []
    for i in range(0,len(split_list)):
        k = split_list[i] in words.words()
        log.append(k)
    ### log contain the logical value for the words    
    ### Number of words per sentence
    df=[]
    for entry in list_of_sentence:
        df.append(len(entry.split()))

    df = pd.DataFrame(df)
    df.columns = ['number_of_words']

    df['index'] = range(0,len(df))

    dat = pd.DataFrame(df.loc[np.repeat(df.index.values,df['number_of_words'])])
    dat['logical'] = log
    dat['log_num'] = dat['logical'].astype(int)

    df['English_words_freq'] = dat.groupby(['index'], as_index = False).sum()['log_num']/df['number_of_words']
    final = pd.DataFrame(list_of_sentence)
    final['logical'] = df['English_words_freq'] &gt; 0.75
    return(final)


Extract_sentence_english(pop)


Does somebody know any better way of doing this ?
Here are some comments on your code. First some stylistic comments, and afterwards some related to the logic of your code.


Method names should follow snake_case – According to the PEP8 standard, your method should be named extract_sentence_english, and slightly better worded, extract_english_sentences
Naming is hard – Most of the names seems to be slightly random. Why log, split_list, df, dat... They are not conveying anything sensible, or indicate how you've processed the original sentences.
Add vertical space – You have some, but could have more vertical space.
Better comments – For example, related to the split_list (renamed to all_words and log (renamed to is_english_words), I would rather write something like this:

# Make a list of all single words based upon the sentences
all_words = [word for sentence in list_of_sentence
                     for word in sentence.split(' ')]

# Make a corresponding list with True if the word is English,
# or False if not
is_english_words = [word in words.words() for word in all_words]


Now the comments explain the contents of the list, and what is happening on the next code lines.  I've also used a list comprehension to create the alternative to log, and with better variable names it is easier to see, and should not as expensive as the original for loop. By the way, range defaults to starting at 0 so it could simply have been range(len(split_list)).

Same kind of logic (and list comprehension) could apply to the creation of df, which could be renamed to sentence_lengths. 
Why all the DataFrame mumble jumble? – I don't really see the need for all the usage of DataFrame (and don't understand what is happening besides the reconnection of words and sentences). More on this in alternate solution below.
Please add space after commas, , – In some of the parameter listings you don't add space after the comma, which makes the code harder to read. Be consistent, and always add space after comma.


Alternative logic

If I've understood your request properly you are getting a whole lot of sentences, and want to return all the English sentences (and loose the French sentences?). You go about this task by removing everything which isn't a-z, splitting into words and checking against a dictionary, and if the English word frequency is above 0.75 you'll return that sentence (without punctuation). 

Doing it that way eliminates on a good indicator of French sentences. French sentences uses loads of accented characters. English doesn't. That is a very good indicator (if the Facebook guys care to write those characters), which you are missing out on.

Your solution also checks against an entire dictionary, which is kind of expensive. A simpler solution would be to keep two smaller list of typical English words and French words, and make sure to remove non-French or non-English words from these lists. I know very little French, but here is an attempt at some trigger word lists:

english_words = ["you", "are", "not", "the", "is", "what", "who", "that"]
french_words = ["le", "ce", "un", "la", "et", "au", "à", "toute"]


The idea is to have enough specific typical words in either list, so that it eases detection of the language. 

So how could this look in an alternate version? Well, here is some code for you to work with:

import re
import random

sample_sentences = ['hi there, how are you mister', 'some none triggering english sentence', 'And another long english statement', 'This is not plain English', 'say','bonjour mister','sauf','tu am      belle','I suis cool', 'Hey YOU', u'Worst trip.ever je, à deteste AirTransat! ce sont les pires', u"Toute personne a droit à l´éducation"]

ENGLISH_WORDS = ["you", "are", "not", "the", "is", "what", "who", "that"]
FRENCH_WORDS = ["le", "ce", "un", "la", "et", "au", u"à", u"é", "toute"]

REQUIRED_NUMBER_OF_WORDS = 1
REQUIRED_ENGLISH_PERCENTAGE = 0.75

def extract_english_sentences(sentences):

  # Compile a pattern of all non-alphanumerics in Unicode
  pattern = re.compile('\W+', re.UNICODE)

  result = []
  print sentences

  # Loop on each sentence, and determine if it English or not
  for sentence in sentences:
    # Get a simplified sentence with only unicode alphanumerics,
    # which are lowercased
    no_punctuation = pattern.sub(' ', sentence).lower()

    # Split into single words
    words_in_sentence = no_punctuation.split()
    number_words = len(words_in_sentence)

    print ("\n&gt;{}&lt;".format(no_punctuation.encode('utf8', 'replace')))

    is_english = None

    # Check whether word is in either trigger list
    for word in words_in_sentence:
      if word in FRENCH_WORDS: 
        print "in french words"
        is_english = False
        break

      if word in ENGLISH_WORDS:
        print "in english words"
        is_english = True
        break

    if is_english is not None and is_english:
      result.append(sentence)
      continue

    # If it didn't match anything in trigger list,
    # and we have more than 5 words, lets check the
    # percentage of English words
    if is_english is None and number_words &gt; REQUIRED_NUMBER_OF_WORDS:
      # Count occurences of each word in the dictionary
      # give by words.words()
      #sum_english_words = sum(word in words.words() 
      #                          for word in words_in_sentence)

      # Don't have dictionary, guess... :-)
      sum_english_words = random.randint(1, number_words)
      print "{} vs {}".format(sum_english_words, number_words)

      if sum_english_words / number_words &gt; REQUIRED_ENGLISH_PERCENTAGE: 
        print "Dictionary says English"
        result.append(sentence)
        continue

    if is_english is not None and not is_english:
      print "French!"
    else:
      print "Couldn't decide"

  return result

print extract_english_sentences(sample_sentences)


As I didn't have any words.words() available, I've commented out that segment, and use a random generator instead. This in combination a lot of print statements should show that in most cases it would continue with next sentence in a lot of the cases, and hopefully not fall through to the dictionary test or "Couldn't decide" very often.

When deployed to production all of the print statements should be removed. Do also note that this code is written in Python 2.7, so no parentheses are needed for the print statements, and some strings needs to be declared as unicode strings. But you didn't state which python version you are using, so I made this in the older variant. Hope this helps you along the way.

Alternate trigger word list tests

At the cost of looping twice through the words_in_sentence list, one could remove the inner for loop, and all other code related to is_english:

# Check whether word is in either trigger list
if any(word in FRENCH_WORDS for word in words_in_sentence):
  print "in french words"
  continue

if any(word in ENGLISH_WORDS for word in words_in_sentence):
  print "in english words"
  result.append(sentence)


Possible major speedup

I just realized that you could get a major speedup related to the dictionary lookup. It is very expensive to get the complete word list, words.words(), for every sentence. This should be done once, and created into a set, which then is used to check for existence within dictionary. 

It's late in the night locally for me, so the following code is untested:

WORDS_SET = set(words.words())

...

sum_english_words = sum(word in WORDS_SET)


This should avoid the repeated loading of the words for every word of every sentence... Will try to test this tomorrow.