title: Greek words recognition while reading url with python
body: <p>I'm new python programmer. I have written a simple script that is doing the following:</p>
  
  <ul>
  <li>asks url from the user</li>
  <li>reads the url (urlopen(url).read())</li>
  <li>tokenizes the result of the above command</li>
  </ul>
  
  <p>I take the result of the tokenization in two files. The one has words in latin characters (english, spanish etc) and the other one the rest (greek words etc). </p>
  
  <p>The problem is that, when I open a greek url, I take the greek from it, but I see it as a sequence of characters, not words (as it happens in the latin case).</p>
  
  <p>I expect to take a list of the words (<code>μαρια</code>, <code>γιωργος</code>, <code>παιδι</code>) (number of items 3) but what i take is <code>('μ','α','ρ','ι', 'α'........)</code> number of items as many as the letters</p>
  
  <p>What should I do? (the encoding is utf-8)</p>
  
  <p>Follows the code:</p>
  
  <pre><code>#!/usr/bin/env python
  # -*- coding: utf-8 -*-
  
  #Importing useful libraries 
  #NOTE: Nltk should be installed first!!!
  import nltk
  import urllib #mporei na einai kai urllib
  import re
  import lxml.html.clean
  import unicodedata
  from urllib import urlopen
  
  http = "http://"
  www = "www."
  #pattern = r'[^\a-z0-9]'
  
  #Demand url from the user
  url=str(raw_input("Please, give a url and then press ENTER: \n"))
  
  
  #Construct a valid url syntax
  if (url.startswith("http://"))==False:
      if(url.startswith("www"))==False:
          msg=str(raw_input("Does it need 'www'? Y/N \n"))
          if (msg=='Y') | (msg=='y'):
              url=http+www+url
          elif (msg=='N') | (msg=='n'):
              url=http+url
          else:
              print "You should type 'y' or 'n'"
      else:
          url=http+url
  
  latin_file = open("Latin_words.txt", "w")
  greek_file = open("Other_chars.txt", "w")
  latin_file.write(url + '\n')
  latin_file.write("The latin words of the above url are the following:" + '\n')
  greek_file.write("Οι ελληνικές λέξεις καθώς και απροσδιόριστοι χαρακτήρες")
  
  #Reading the given url
  
  raw=urllib.urlopen(url).read()
  
  #Retrieve the html body from the url. Clean it from html special characters
  pure = nltk.clean_html(raw)
  text = pure
  
  #Retrieve the words (tokens) of the html body in a list
  tokens = nltk.word_tokenize(text)
  
  counter=0
  greeks=0
  for i in tokens:
      if re.search('[^a-zA-Z]', i):
          #greeks+=1
          greek_file.write(i)
      else:
          if len(i)&gt;=4:
              print i
              counter+=1
              latin_file.write(i + '\n')
          else:
              del i
  
  
  #Print the number of words that I shall take as a result
  print "The number of latin tokens is: %d" %counter
  
  latin_file.write("The number of latin tokens is: %d and the number of other characters is: %d" %(counter, greeks))
  latin_file.close()
  greek_file.close()
  </code></pre>
  
  <p>I checked it in many ways, and, as far as I can get it, the program just recognises greek characters but fails to recognise greek words, meaning, the space with witch we seperate words!</p>
  
  <p>If i type greek sentences with space in my terminal, it appears correctly. The problem occurs when i read something (like body from html page)</p>
  
  <p>In addition, in text_file.write(i), regarding the greek i's, if i write text_file.write(i+ '\n'), the result is unidentified characters, aka, i lose my encoding!</p>
  
  <p>Any ideas concerning the above?</p>