from nltk.corpus import stopwords
import string
from transform.normalizer import *
from transform.parser import *
from match.match import *
import inspect
import jellyfish
from retrieve.search import *

"""
Explanation: Address Normalization
End of explanation
"""


punctuation = set(string.punctuation)
language = 'portuguese'

prefix_file = '../data/prefixes.csv'

with open(prefix_file, 'r') as g:
    prefixes = g.read().splitlines()
address_prefixes = prefixes

stopw = stopwords.words(language)

"""
Explanation: First, let's read the data that we're going to use to normalize and parse the addresses:
End of explanation
"""


address = "Rua XV de Novembro, 123 bloco 23 A"

normalized_address = normalize_address(
    address, punctuation, stopw , address_prefixes)
print("Normalized address: ", normalized_address)

"""
Explanation: punctuation is the file with the punctuation characters that we want to remove.
prefixes are the the common address prefixes that we want to remove.
stopw are the common Portuguese stopwords that we also want to remove.
End of explanation
"""


inspect.getsourcelines(normalize_address)

"""
Explanation: So what are we doing here? Let's see what normalize_address is doing:
End of explanation
"""


parsed_address = parse_address(normalized_address)
print(parsed_address)

"""
Explanation: So we are doing several operations in sequence:

transform_encoding
transform_case
remove_punctuation
remove_stopwords
remove_address_prefixes

Applying the next function to the results of the previous one. So what's next?
Parsing the address
After we normalized the address we want to parse it, selecting the relevant parts. We can do that with Regex or Named Entity Recognition. First, let's try to use regular expressions:
End of explanation
"""


inspect.getsourcelines(parse_address)

"""
Explanation: So how are we doing that?
End of explanation
"""


schema = create_schema()
idx = create_index(schema, 'indexdir')
results = search(parsed_address['street'], 'street', idx)
print(results)

"""
Explanation: That's the regular expression: ^(\\S+\\D*?)\\s*(\\d+)|(\\S.*).
It means:
```
1st Alternative: ^(\S+\D?)\s(\d+)
^ assert position at start of the string
1st Capturing group (\S+\D?)
\S+ match any non-white space character [^\r\n\t\f ]
Quantifier: + Between one and unlimited times, as many times as possible, giving back as needed [greedy]
\D? match any character that's not a digit [^0-9]
Quantifier: ? Between zero and unlimited times, as few times as possible, expanding as needed [lazy]
\s match any white space character [\r\n\t\f ]
Quantifier: * Between zero and unlimited times, as many times as possible, giving back as needed [greedy]
2nd Capturing group (\d+)
\d+ match a digit [0-9]
Quantifier: + Between one and unlimited times, as many times as possible, giving back as needed [greedy]
2nd Alternative: (\S.*)
3rd Capturing group (\S.)
\S match any non-white space character [^\r\n\t\f ]
. matches any character (except newline)
Quantifier: * Between zero and unlimited times, as many times as possible, giving back as needed [greedy]
```

Wow, that's very hard to understand. 
But now we have our address normalized and separated on its components. We can now try to match it with the canonical source. 
Match
The previous steps did not correct for misspellings or other errors. If we have a canonical database, we can try to reduce those errors and transform our address in our database to a canonical form. For that, we have to match the normalized address with our reference database. First, how we calculate that two addresses are similar?
Similarity
We can compute a similarity between two strings. There are several algorithms to do that. We will use the Jaro-Winkler distance for that. There are several others, like 

Cosine Distance
Hamming Distance

Candidates for match
How can we retrieve candidates to match from our canonical database?
There are a few approaches:

Brute force (all against all)
Search by field

We will try to search for candidates and do a match with them.
Search
End of explanation
"""


print(address)

"""
Explanation: So now we have our candidates! But the other informations are different from what whe have in our address. Is this a match?
Match
Let's devise a way to match these two addresses:
End of explanation
"""


similarity(parsed_address['street'],results[0]['street'] )

similarity(parsed_address['street'],results[1]['street'] )

"""
Explanation: We have some prior information about how addresses are and which parts are more important than others. We can devise a matching algorithm with a linear regression, for example, using the knowledge that street names are more important than complements:
End of explanation
"""


similarity(str(parsed_address['number']),str(results[0]['number'] ))

similarity(str(parsed_address['number']),str(results[1]['number'] ))

"""
Explanation: So the similarity of the street name is exactly the same. Let's compare the numbers:
End of explanation
"""


similarity(parsed_address['complement'],results[0]['complement'] )

similarity(parsed_address['complement'],results[1]['complement'] )

"""
Explanation: Ops, still the same. Let's go to complements:
End of explanation
"""


print("Original Address:", address)
print("Canonical Address:", str(results[1]['street']) + ', ' + str(results[1]['number']) + ' ' + str(results[1]['complement']))

"""
Explanation: Ok, there's a small difference, but we will assume that we can work with that! The second canonical address is a better match than the first one:
End of explanation
"""


