"""
# Better NLP
"""
"""
This is a wrapper program/library that encapsulates a couple of NLP libraries that are popular among the AI and ML communities.

Examples have been used to illustrate the usage as much as possible. Not all the APIs of the underlying libraries have been covered.

The idea is to keep the API language as high-level as possible, so its easier to use and stays human-readable.

**One of the key differentiators of this library compared to the others are that you get metadata returned to you by each API function about the duration of execution and other details about the execution.**

Libraries / frameworks covered:

- SpaCy ([site](https://spacy.io/) | [docs](https://spacy.io/usage/))
- Textacy ([github](https://github.com/chartbeat-labs/textacy) | [docs](https://chartbeat-labs.github.io/textacy/))

See [https://github.com/neomatrix369/awesome-ai-ml-dl/blob/master/examples/better-nlp](https://github.com/neomatrix369/awesome-ai-ml-dl/blob/master/examples/better-nlp) for more details.

#### Kaggle kernel script: https://www.kaggle.com/neomatrix369/betternlpclass

### This notebook will demonstrate the below NLP features / functionalities, using the above mentioned libraries

- Extract entities
- Noun extraction
- Gather facts
- Obfuscate privacy details
- Parts-of-speech
"""
"""
## License

 Copyright 2019 Mani Sarkar

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.

 ### Please feel free to fork this work as long as you maintain the above license and provide citation.
"""
"""
#### Setup and installation ( optional )

In case, this notebook is running in a local environment (Linux/MacOS) or _Google Colab_ environment and in case it does not have the necessary dependencies installed then please execute the steps in the next section.

Otherwise, please SKIP to the **Install Spacy model ( NOT optional )** section.
"""
print("Installing / updating python packages using pip")
! apt-get update && apt-get install -y --fix-missing liblapack-dev libswscale-dev pkg-config
"""
#### Install Spacy model ( NOT optional )

Install the large English language model for spaCy - will be needed for the examples in this notebooks.

**Note:** from observation it appears that spaCy model should be installed towards the end of the installation process, it avoid errors when running programs using the model.
"""
%%time
%%bash

python -m spacy download en_core_web_lg
python -m spacy link en_core_web_lg en || true
"""
## Examples
"""
"""
### Extract entities
"""
### Note: this take a bit of time to load the first time, as it loads the model, etc..
### Subsequent runs are faster as it lazy initialises model related variable(s) and just loads it from cache
from better_nlp_class import BetterNLP
betterNLP = BetterNLP()
# Can be any factual text or any text to experiment with
generic_text = "Denis Guedj (1940 – April 24, 2010) was a French novelist and a professor of the History of Science at Paris VIII University. He was born in Setif. He spent many years devising courses and games to teach adults and children math. He is the author of Numbers: The Universal Language and of the novel The Parrot's Theorem. He died in Paris. "
model_loading_result = betterNLP.load_nlp_model()
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
print("model_loading_time_in_secs=",model_loading_result['model_loading_time_in_secs'])
print("model_loading_method=",model_loading_result['model_loading_method'])
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")

model = model_loading_result["model"]
extracted_entities = betterNLP.extract_entities(model, generic_text)

print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
print("extract_entities_processing_time_in_secs=", extracted_entities['extract_entities_processing_time_in_secs'])
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
betterNLP.pretty_print(extracted_entities["extracted_entities"])
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
betterNLP.pretty_print(betterNLP.token_entity_types())
"""
### Noun extraction
"""
chunks = betterNLP.extract_noun_chunks(model, generic_text)
chunks = chunks.get("noun_chunks")

print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
betterNLP.pretty_print(chunks)
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
"""
### Gather facts
"""
target_topic = "Denis Guedj"
extracted_facts = betterNLP.extract_facts(model, generic_text, target_topic)

print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
betterNLP.pretty_print(extracted_facts.get("facts"))
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
"""
### Obfuscate privacy details
"""
obfuscated_text = betterNLP.obfuscate_text(model, generic_text)
obfuscated_text = obfuscated_text.get("obfuscated_text")
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
print("Obfuscated generic text: ", "".join(obfuscated_text))
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
"""
### Parts-of-speech
"""
generic_text = u'Apple is looking at buying U.K. startup for $1 billion'
parts_of_speech_tagging = betterNLP.parts_of_speech_tagging(model, generic_text).get("parts_of_speech")
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
betterNLP.pretty_print(parts_of_speech_tagging)
print("~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~")
### Linting betternlpclass
!pip install --upgrade pylint pyqt5==5.12 pyqtwebengine==5.12 &> /dev/null || true 
!pylint better_nlp_class