%%capture
!wget https://github.com/cltl/python-for-text-analysis/raw/master/zips/Data.zip
!wget https://github.com/cltl/python-for-text-analysis/raw/master/zips/images.zip
!wget https://github.com/cltl/python-for-text-analysis/raw/master/zips/Extra_Material.zip

!unzip Data.zip -d ../
!unzip images.zip -d ./
!unzip Extra_Material.zip -d ../

!rm Data.zip
!rm Extra_Material.zip
!rm images.zip

"""
Explanation: <a href="https://colab.research.google.com/github/cltl/python-for-text-analysis/blob/colab/Assignments-colab/ASSIGNMENT_4a.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
End of explanation
"""


def read_csv(input_file, delimiter=","):
    # your code here


# test your function here
filename = "../Data/csv_data/trump_facebook.tsv"
status_updates = read_csv(filename, delimiter="\t") 
status_updates[0:2]

"""
Explanation: Assignment 4a: Data structures (CSV/TSV and JSON)
Deadline for Assignment 4a+b: Friday, October 15, 2021 (14.30) via Canvas (Assignment 4) 

Please name your notebook with the following naming convention: 
  ASSIGNMENT_4a_FIRSTNAME_LASTNAME.ipynb 
Please submit your complete assignment (4a + 4b) by compressing all your material into a single .zip file following this naming convention: ASSIGNMENT_4_FIRSTNAME_LASTNAME.zip.  

Please note that there is a BA and an MA version of Assignment 4b
In case you are not sure about creating a zip file from a folder, please refer to this guide (or any other guide you find online).
If you have questions about this chapter, please contact us at cltl.python.course@gmail.com. Questions and answers will be collected in this Q&A document, so please check if your question has already been answered. 
In this block, we covered the following chapters about data formats:

Chapter 16 - Data Formats I (CSV/TSV)
Chapter 17 - Data Formats II (JSON)
Chapter 18 - Data Formats III (XML) only for master-level course

In this assignment, you will also have to apply your knowledge about containers. If you get stuck, you are likely to find solutions in the chapters about containers (Block 2). 
Tip:
It could happen that your code throws a unicode error when you're trying to open one of the files used in this assignment. If this is the case, you can probably solve if by specifying the encoding when reading in the file:
python
with open(your/file/path, 'r', encoding = 'utf-8') as infile:
    #your code
Exercise 1: Trump's Facebook Status Updates (CSV/TSV)
In the folder ../Data/csv_data there is a TSV file called trump_facebook.tsv that contains Facebook status updates posted by Donald Trump. It was downloaded from here. Follow the instructions below to read the file and find specific status updates.
1a. Write your own function for reading CSV
Write a function called read_csv() that has two parameters: 

input_file (positional parameter) and 
delimiter (keyword parameter with default string ","). 

The function should read the file and return status_updates which contains the content of the file as a 'list of dicts'. When tested on ../Data/Trump-Facebook/FacebookStatuses.tsv the first two status updates should thus be represented as follows:
[{'link_name': 'Timeline Photos',
  'num_angrys': '7',
  'num_comments': '543',
  'num_hahas': '17',
  'num_likes': '6178',
  'num_loves': '572',
  'num_reactions': '6813',
  'num_sads': '0',
  'num_shares': '359',
  'num_wows': '39',
  'status_id': '153080620724_10157915294545725',
  'status_link': 'https://www.facebook.com/DonaldTrump/photos/a.488852220724.393301.153080620724/10157915294545725/?type=3',
  'status_message': 'Beautiful evening in Wisconsin- THANK YOU for your incredible support tonight! Everyone get out on November 8th - and VOTE! LETS MAKE AMERICA GREAT AGAIN! -DJT',
  'status_published': '10/17/2016 20:56:51',
  'status_type': 'photo'},
 {'link_name': '',
  'num_angrys': '5211',
  'num_comments': '3644',
  'num_hahas': '75',
  'num_likes': '26649',
  'num_loves': '487',
  'num_reactions': '33768',
  'num_sads': '191',
  'num_shares': '17653',
  'num_wows': '1155',
  'status_id': '153080620724_10157914483265725',
  'status_link': 'https://www.facebook.com/DonaldTrump/videos/10157914483265725/',
  'status_message': "The State Department's quid pro quo scheme proves how CORRUPT our system is. Attempting to protect Crooked Hillary, NOT our American service members or national security information, is absolutely DISGRACEFUL. The American people deserve so much better. On November 8th, we will END this RIGGED system once and for all!",
  'status_published': '10/17/2016 18:00:41',
  'status_type': 'video'}]
DO NOT USE THE CSV MODULE FOR THIS EXERCISE!
End of explanation
"""


import csv

filename = "../Data/csv_data/trump_facebook.tsv"
with open(filename, "r") as infile:
    status_updates = []
    csv_reader = csv.DictReader(infile, delimiter='\t')
    for row in csv_reader:
        status_updates.append(row)
status_updates[0:2]

"""
Explanation: In case you didn't manage to create the read_csv() function, run the following code using the DictReader() method from the csv module to get the data in the right format for the following exercises:
End of explanation
"""


def get_update_most_responded_to():
    # your code here

# test your function here

"""
Explanation: 1b. Find the status updates with the most responses
Define a function called get_update_most_responded_to() that has the following parameters: 
* status_updates (positional parameter) 
* response_type (keyword parameter with default string "likes") 
The fuction should find the status update that received the highest number of possible reactions to a Facebook status (emoji such as 'angrys', 'comments', 'hahas', etc. - anything that starts with 'num_'). It should return three strings: the status_message, the status_type and the status_link of this particular status update.
End of explanation
"""


def get_longest_update():
    # your code here

# test your function here

"""
Explanation: 1c. Find the longest status updates
Define a function called get_longest_update() that has the following parameters: 
* status_updates (positional parameter) 
* length_type (keyword parameter with default string "tokens"). 
The function should find the longest update. By default, the fuction should find the status update that is the longest in terms of number of tokens. Also implement the options to find the longst status update in terms of characters or sentences in the message. These options should be carried out when length_type is changed to "sentences" or "characters" 
The function should return the status message (called 'status_message' in the data structure) of the longest update as a string. 
Attention: It is recommended to use NLTK for this exercise.
End of explanation
"""


def get_update_with_keywords():
    # your code here

keywords = ["clinton", "obama"] # test with these keywords; also experiment with other keywords
# test your function here

"""
Explanation: 1d. Find the status updates containing specific keywords
Define a function called get_updates_with_keywords() that takes three input arguments: 

status_updates (mandatory positional argument) 
keywords (mandatory positional argument) 
case_sensitive (keyword argument with default False)

The fuction should find the status updates that contain any of the keywords. The parameter case_sensitive should specify whether uppercase and lowercase characters must be treated as distinct. 
The function should return filtered_status_updates, which is a list of dictioaries with all information about the status updates (same format as the input argument 'status_updates'). 
Attention: It is highly recommended to use NLTK for this exercise. Make sure that you tokenize the messages before you look for keywords.
End of explanation
"""


import requests

# Download data on prizes
api_url = "http://api.nobelprize.org/v1/prize.json"
r = requests.get(api_url)
dict_prizes = r.json()
# uncomment the line below if you'd like to see what's inside dict_prizes
#dict_prizes 

# Download data on laureates
api_url = "http://api.nobelprize.org/v1/laureate.json"
r = requests.get(api_url)
dict_laureates = r.json()
# uncomment the line below if you'd like to see what's inside dict_prizes
#dict_laureates 

"""
Explanation: Exercise 2: Nobel Prize Winners (JSON)
There is a lot of interesting data online. For example, the Nobel Prize Organisaton provides the Nobel Prize API that allows you to download information about the prizes, the laureates and the countries. 
The information is formatted in JSON. Have a look at the following URLs:
- http://api.nobelprize.org/v1/prize.json
- http://api.nobelprize.org/v1/laureate.json
- http://api.nobelprize.org/v1/country.json
For this exercise, we will only look at the prizes and the laureates. 
We can download the data using the requests module. How this works is shown below.
End of explanation
"""


# load laureates.json and prize.json here

"""
Explanation: 2a. Read the JSON files
We have already stored the data as the JSON files laureate.json and prize.json in the folder ../Data/json_data/NobelPrize. Open these JSON files and load them as the Python dictionaries dict_laureates and dict_prizes.
End of explanation
"""


def get_laureates():
    # your code here                     


year = 2018
category = "peace"
# test your function here

"""
Explanation: 2b. Get all laureates from year and category
Create a function called get_laureates() that thas three parameters: 

dict_prizes (positional parameter) 
year (keyword parameter with default None) 
category (keyword parameter with default None) 

The function should find all laureates that received the Nobel Prize, optionally in a specific year and/or category (specified using the keywords year and category). It should return a list of the full names of the laureates.
For example, for the year 2018 and category "peace" it should return the list ['Denis Mukwege', 'Nadia Murad'].
End of explanation
"""


def get_affiliation_prizes():
    # your code here

# test your function here

"""
Explanation: 2c. Get all prizes from affiliations
Create a function called get_affiliation_prizes() that takes one input parameters: 

dict_laureates (positional parameter) 

The function should find all affiliations that were involved in winning the Nobel Prize and provide information on the category and year of those Nobel Prizes. It should return a nested dictionary of the following format:
{
    "A.F. Ioffe Physico-Technical Institute": [
        {"category": "physics", "year": "2000"}
    ],
    "Aarhus University": [
        {"category": "chemistry", "year": "1997"},
        {"category": "economics","year": "2010"}
    ]
}
Tip: some of the entries will lack information (for example, there is no associated affiliation). Use if-statements to check if essential information is present. 
General tip for working with data: If your code breaks, check whether your assumptions about the data hold (very often, they unfortunatelydo not). For instance, a dictionary key you thought was always present is missing from a couple of dictionaries, etc.
End of explanation
"""


# write the resulting dictionary to 'json_file'

"""
Explanation: 2d. Write to JSON
Next, write the dictionary created in the previous exercise to a JSON file using the following path: 
../Data/json_data/NobelPrize/nobel_prizes_affiliations.json.
End of explanation
"""


