%%capture
!wget https://github.com/cltl/python-for-text-analysis/raw/master/zips/Data.zip
!wget https://github.com/cltl/python-for-text-analysis/raw/master/zips/images.zip
!wget https://github.com/cltl/python-for-text-analysis/raw/master/zips/Extra_Material.zip

!unzip Data.zip -d ../
!unzip images.zip -d ./
!unzip Extra_Material.zip -d ../

!rm Data.zip
!rm Extra_Material.zip
!rm images.zip


"""
Explanation: <a href="https://colab.research.google.com/github/cltl/python-for-text-analysis/blob/colab/Assignments-colab/ASSIGNMENT_3b.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
End of explanation
"""


# Downloading data - you get this for free :-) 

import requests
import os


def download_book(url):
    """
    Download book given a url to a book in .txt format and return it as a string.
    """
    text_request = requests.get(url)
    text = text_request.text
    return text


book_urls = dict()
book_urls['HuckFinn'] = 'http://www.gutenberg.org/cache/epub/7100/pg7100.txt'
book_urls['Macbeth'] = 'http://www.gutenberg.org/cache/epub/1533/pg1533.txt'
book_urls['AnnaKarenina'] = 'http://www.gutenberg.org/files/1399/1399-0.txt'


if not os.path.isdir('../Data/books/'):
    os.mkdir('../Data/books/')
    
    
for name, url in book_urls.items():
    text = download_book(url)
    with open('../Data/books/'+name+'.txt', 'w', encoding='utf-8') as outfile:
        outfile.write(text)

"""
Explanation: Assignment 3b: Writing your own nlp program
Due: Friday the 1st of October 2021 14:30.


Please submit your assignment (notebooks of parts 3a and 3b + additional files) as a single .zip file using Canvas (Assignments --> Assignment 3)


Please name your zip file with the following naming convention: ASSIGNMENT_3_FIRSTNAME_LASTNAME.zip


IMPORTANTE NOTE:
* The students who follow the Bachelor version of this course, i.e., the course Introduction to Python for Humanities and Social Sciences (L_AABAALG075) as part of the minor Digital Humanities, do not have to do Exercises 3 and 4 of Assignment 3b
* The other students, i.e., who follow the Master version of  course, which is Programming in Python for Text Analysis (L_AAMPLIN021), are required to do Exercises 3 and 4 of Assignment 3b
If you have questions about this topic, please contact us (cltl.python.course@gmail.com). Questions and answers will be collected on Piazza, so please check if your question has already been answered first.
In this part of the assignment, we will carry out our own little text analysis project. The goal is to gain some insights into longer texts without having to read them all in detail. 
This part of the assignment builds on some notions that have been revised in part A of the assignment. Please feel free to go back to part A and reuse your code whenever possible. 
The goals of this part are:

divide a problem into smaller sub-problems and test code using small examples
doing text analysis and writing results to a file
combining small functions into bigger functions

Tip: The assignment is split into four steps, which are divided into smaller steps. Instead of doing everything step by step, we highly recommend you read all sub-steps of a step first and then start coding. In many cases, the sub-steps are there to help you split the problem into manageable sub-problems, but it is still good to keep the overall goal in mind. 
Preparation: Data collection
In the directory ../Data/books/, you should find three .txt files. If not, you can use the following cell to download them. Also, feel free to look at the code to learn how to download .txt files from the web. 
We defined a function called download_book which downloads a book in .txt format. Then we define a dictionary with names and urls. We loop through the dictionary, download each book and write it to a file stored in the directory books in the current working directory. You don't need to do anything - just run the cell and the files will be downloaded to your computer.
End of explanation
"""


import nltk
nltk.download('punkt')

from nltk.tokenize import sent_tokenize, word_tokenize

text = 'Python is a programming language. It was created by Guido van Rossum.'
for sent in sent_tokenize(text):
    print('SENTENCE', sent)
    tokens = word_tokenize(sent)
    print('TOKENS', tokens)

"""
Explanation: Encoding issues with txt files
For Windows users, the file “AnnaKarenina.txt” gets the encoding cp1252. 
In order to open the file, you have to add encoding='utf-8', i.e.,
python
a_path = 'some path on your computer.txt'
with open(a_path, mode='r', encoding='utf-8'):
    # process file
Exercise 1
Was the download successful? Let's start writing code! Please create the following two Python modules:

Python module analyze.py This module you will call from the command line
Python module utils.py This module will contain your helper functions

More precisely, please create two files, analyze.py and utils.py, which are both placed in the same directory as this notebook. The two files are empty at this stage of the assignment.
1.a) Write a function called get_paths and store it in the Python module utils.py
The function get_paths:
* takes one positional parameter called input_folder
* the function stores all paths to .txt files in the input_folder in a list
* the function returns a list of strings, i.e., each string is a file path
Once you've created the function and stored it in utils.py:
* Import the function into analyze.py, using from utils import get_paths
* Call the function inside analyze.py (input_folder="../Data/books")
* Assign the output of the function to a variable and print this variable.
* call analyze.py from the command line to test it
Exercise 2
2.a) Let's get a little bit of an overview of what we can find in each text. Write a function called get_basic_stats.
The function get_basic_stats:
* has one positional parameter called txt_path which is the path to a txt file
* reads the content of the txt file into a string
* Computes the following statistics:
    * The number of sentences
    * The number of tokens
    * The size of the vocabulary used (i.e. unique tokens)
    * the number of chapters/acts:
        * count occurrences of 'CHAPTER' in HuckFinn.txt
        * count occurrences of 'Chapter ' (with the space) in AnnaKarenina.txt
        * count occurrences of 'ACT' in Macbeth.txt
* return a dictionary with four key:value pairs, one for each statistic described above:
    * num_sents
    * num_tokens
    * vocab_size 
    * num_chapters_or_acts
In order to compute the statistics, you need to perform sentence splitting and tokenization. Here is an example snippet.
End of explanation
"""


import os 
basename = os.path.basename('../Data/books/HuckFinn.txt')
book = basename.strip('.txt')
print(book)

"""
Explanation: 2.b) Store the function in the Python module utils.py. Import it in analyze.py. 
Edit analyze.py so that:
* you first call the function get_paths
* create an empty dictionary called book2stats, i.e., book2stats = {}
* Loop over the list of txt files (the output from get_paths) and call the function get_basic_stats on each file
* print the output of calling the function get_basic_stats on each file.
* update the dictionary book2stats with each iteration of the for loop.
Tip: book2stats is a dictionary mapping a book name (the key), e.g., ‘AnnaKarenina’, to a dictionary (the value) (the output from get_basic_stats)
Tip: please use the following code snippet to obtain the basename name of a file path:
End of explanation
"""


import operator 

token2freq = {'a': 1000, 'the': 100, 'cow' : 5}
for token, freq in sorted(token2freq.items(), 
                          key=operator.itemgetter(1),
                          reverse=True):
    print(token, freq)

"""
Explanation: Please note that Exercises 3 and 4, respectively, are designed to be difficult. You will have to combine what you have learnt so far to complete these exercises.
Exercise 3
Let's compare the books based on the statistics. Create a dictionary stats2book_with_highest_value in analyze.py with four keys:
* num_sents
* num_tokens
* vocab_size 
* num_chapters_or_acts
The values are not the frequencies, but the book that has the highest value for the statistic. Make use of the book2stats dictionary to accomplish this.
Exercise 4
4.a) The statistics above already provide some insights, but we want to know a bit more about what the books are about. To do this, we want to get the 30 most frequent tokens of each book. Edit the function get_basic_stats to add one more key:value pair:
* the key is top_30_tokens
* the value is a list of the 30 most frequent words in the text.
4.b) Write the top 30 tokens (one on each line) for each file to disk using the naming top_30_[FILENAME]:
* top_30_AnnaKarenina.txt
* top_30_HuckFinn.txt
* top_30_Macbeth.txt
Example of file (the and and may not be the most frequent tokens, these are just examples):
the 
and 
..
The following code snippet can help you with obtaining the top 30 occurring tokens. The goal is to call the function you updated in Exercise 4a, i.e., get_basic_stats, in the file analyze.py. This also makes it possible to write the top 30 tokens to files.
End of explanation
"""


