title: Spacy - NLTK: Language detection
body: <p>I am currently working on a project dealing with a bunch of social media posts.
  Some of these posts are in English and some in Spanish. </p>
  
  <p>My current code runs quite smoothly. However, I am asking myself does Spacy/NLTK automatically detect which language stemmer/stopwords/etc.  it has to use for each post (depending on whether it is an English or Spanish post)? At the moment, I am just parsing each post to a stemmer without explicitly specifying the language.  </p>
  
  <p>This is a snippet of my current script:</p>
  
  <pre><code>import re
  import pandas as pd
  !pip install pyphen
  import pyphen
  !pip install spacy
  import spacy
  !pip install nltk
  import nltk
  from nltk import SnowballStemmer
  from nltk.corpus import stopwords
  from nltk.stem.porter import PorterStemmer
  !pip install spacy-langdetect
  from spacy_langdetect import LanguageDetector
  !pip install textblob
  from textblob import TextBlob
  
  # Download Stopwords
  nltk.download('stopwords')
  stop_words_eng = set(stopwords.words('english'))
  stop_words_es = set(stopwords.words('spanish'))
  
  # Import Stemmer
  p_stemmer = PorterStemmer()    
  #Snowball (Porter2): Nearly universally regarded as an improvement over porter, and for good reason. 
  snowball_stemmer = SnowballStemmer("english")
  dic = pyphen.Pyphen(lang='en')
  
  # Load Data
  data = pd.read_csv("mergerfile.csv", error_bad_lines=False)
  pd.set_option('display.max_columns', None)
  posts = data.loc[data["ad_creative"] != "NONE"]
  
  # Functions
  def get_number_of_sentences(text):
      sentences = [sent.string.strip() for sent in text.sents]
  
      return len(sentences)
  
  
  def get_average_sentence_length(text):
      number_of_sentences = get_number_of_sentences(text)
      tokens = [token.text for token in text]
  
      return len(tokens) / number_of_sentences
  
  
  def get_token_length(text):
      tokens = [token.text for token in text]
  
      return len(tokens)
  
  def text_analyzer(data_frame):
      content = []
      label = []
      avg_sentence_length = []
      number_sentences = []
      number_words = []
  
     for string in data_frame:
         string.join("")
  
         if len(string) &lt;= 4:
             print(string)
             print("filtered")
             content.append(string)
             avg_sentence_length.append("filtered")
             number_sentences.append("filtered")
             number_words.append("filtered")
  
         else:
             # print list
             print(string)
             content.append(string)
             ##Average Sentence Lenght
             result = get_average_sentence_length(nlp(string))
             avg_sentence_length.append(result)
             print("avg sentence length:", result)
  
             ##Number of Sentences
             result = get_number_of_sentences(nlp(string))
             number_sentences.append(result)
             print("#sentences:", result)
  
             ##Number of words
             result = get_token_length(nlp(string))
             number_words.append(result)
             print("#Words", result)
  
  content, avg_sentence_length, number_sentences, number_words = text_analyzer(
      data["posts"])
  </code></pre>