import wikinetworking as wn
import networkx as nx
from pyquery import PyQuery
%matplotlib inline

print "OK"

"""
Explanation: Wiki Networking - Extended Example
Introduction
Network graphs consisting of nodes and edges and be used to visualize the relationships between people. The results can inform a viewer of groups of relationships. This notebook has examples for all functions available and a variety of text mining examples. It demonstrates how to crawl Wiki articles and represent the links between these articles as a graph.
import statements
First, we will import a few necessary packages, including wikinetworking from this repository. It includes functions for crawling and text mining Wiki data as well as graphing the result of the crawl.
End of explanation
"""


iron_man_page = PyQuery(url="https://en.wikipedia.org/wiki/Iron_man")
iron_man_links = wn.filter_links(iron_man_page)
print len(iron_man_links), "links retrieved"

"""
Explanation: filter_links
This function accepts a PyQuery object and returns a list of Wikipedia article links from the article's main body text. It will not return links that are redirects or links that are anchors within other pages. Optionally, you can specify a DOM selector for the type of element containing links you wish to retrieve.
In this example, we create a PyQuery object for the Iron Man article on Wikipedia, retrieve its links with filter_links and output the number of links we retrieved.
End of explanation
"""


hip_hop_page = PyQuery(url="https://en.wikipedia.org/wiki/List_of_hip_hop_musicians")
hip_hop_links = wn.filter_links(hip_hop_page, "li")
print len(hip_hop_links), "links retrieved"

"""
Explanation: In this example, we retrieve links from a list of Hip Hop musicians. Notice that each link is contained inside of a li list item HTML tag.
End of explanation
"""


harry_potter_page = PyQuery(url="https://en.wikipedia.org/wiki/List_of_Harry_Potter_characters")
harry_potter_links = wn.filter_links(harry_potter_page, "li")
print len(harry_potter_links), "links retrieved"

"""
Explanation: In this example, we do the same thing for Harry Potter Characters. Notice that many links are omitted - these links are redirects or links to sections within larger articles.
End of explanation
"""


marvel_d_page = PyQuery(url="https://en.wikipedia.org/wiki/List_of_Marvel_Comics_characters:_D")
marvel_d_links = wn.filter_links(marvel_d_page, ".hatnote")
print marvel_d_links

"""
Explanation: In this example, we are retrieving a list of Marvel Comics Characters beginning with the letter D. In this case, we specify the DOM selector .hatnote. This is because we only want links for Marvel characters that have a dedicated Main article. The Main article elements all have the .hatnote CSS class, so it is an easy way of selecting these articles. In this example, we simply print the contents of the list.
End of explanation
"""


sections = [letter for letter in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ']
sections.append('0-9')
urls = ["https://en.wikipedia.org/wiki/List_of_Marvel_Comics_characters:_" + section for section in sections]
print "URLs:", urls

character_links = wn.retrieve_multipage(urls, ".hatnote", True)
print len(character_links), "links retrieved"

"""
Explanation: retrieve_multipage
This function retrieves data from a list of URLs. This is useful for when data on a Wikipedia category is broken up into multiple individual pages.
In this example, we are retrieving lists from all subpages from the List of Marvel Comics Characters. The URL pattern for these articles is https://en.wikipedia.org/wiki/List_of_Marvel_Comics_characters:_ followed by a section name.
End of explanation
"""


wn.write_list(character_links, "all_marvel_chars.txt")
print len(wn.read_list("all_marvel_chars.txt")), "read"

"""
Explanation: write_list and read_list
These are convenience functions for writing and reading list data.
In this example, we write all of the Marvel character links we retrieved to a text file, and verify that we can read the data from the file.
End of explanation
"""


hip_hop_page = PyQuery(url="https://en.wikipedia.org/wiki/List_of_hip_hop_musicians")
hip_hop_links = wn.filter_links(hip_hop_page, "li")
print len(hip_hop_links), "links retrieved from List of Hip Hop Musicians"

bet_hip_hop_awards_page = PyQuery(url="https://en.wikipedia.org/wiki/BET_Hip_Hop_Awards")
bet_hip_hop_awards_links = wn.filter_links(bet_hip_hop_awards_page, "li")
print len(bet_hip_hop_awards_links), "links retrieved from BET Hip Hop Awards"

award_winner_links = wn.intersection(hip_hop_links, bet_hip_hop_awards_links)

print "BET Hip Hop Award winners:", award_winner_links

"""
Explanation: intersection
Returns a list of elements that appear in both provided lists
The intersection function is useful for cross referencing links from one Wikipedia page with another. For example, the List of Hip Hop Musicians contains many musicians, but we may not have the time or resources to crawl every single artist. The BET Hip Hop Awards page has links to artists that have won an award and may be significant for our search, but it also contains links to songs and other media that may not be articles about individual Hip Hop artists. In addition, each Wiki page will have links to completely unrelated articles. By taking the intersection of both, we get a list containing links of only Hip Hop arists that have won a BET Hip Hop Award.
End of explanation
"""


bet_winner_crawl = wn.crawl("/wiki/Jadakiss", accept=award_winner_links)

print bet_winner_crawl

"""
Explanation: crawl
This crawls articles iteratively, with a two second delay between each page retrieval. It requires a starting URL fragment and an accept list of URLs the crawler should follow. It returns a dictionary structure with each URL fragment as a key and crawl data (links, title and depth) as a value.
In this example, we start a crawl at the article for Jadakiss. As the accept list of URLs that the crawler is allowed to follow, we will use the award_winner_links list from earlier. This isolates the crawl to only Artist winners of BET Hip Hop Awards. Therefore, our social network will be a network built of BET Hip Hop Award Winners and stored in the bet_winner_crawl dictionary.
End of explanation
"""


bet_directed_graph_data = wn.directed_graph(bet_winner_crawl)
print bet_directed_graph_data

bet_undirected_graph_data = wn.undirected_graph(bet_winner_crawl)
print bet_undirected_graph_data

"""
Explanation: directed_graph and undirected_graph
These functions flatten the raw crawl data into something a little more usable. It returns a dictionary with only article title names as the key, and corresponding urls and a dictionary of edges and weights to each article. The directed_graph function allows each article to retain a separate edge representing back links, while the undirected_graph creates one edge from each pair of article where the weight is a sum of the links between each article. For example, directed_graph might produce:
{
   'Iron Man' : {
       'url' : '/wiki/Iron_Man',
       'edges' : {
           'Captain America' : 11
       }
   },
   'Captain America' : {
       'url' : '/wiki/Captain_America',
       'edges' : {
           'Iron Man' : 8
       }
   }
}
...while the same data passed to undirected_graph might produce:
{
    'Iron Man' : {
        'url' : '/wiki/Iron_Man',
        'edges' : {
            'Captain America' : 19
        }
    },
    'Captain America' : {
        'url' : /wiki/Captain_America',
        'edges' : {
        }
    }
}
End of explanation
"""


wn.save_dict(bet_undirected_graph_data, "bet_network.json")
print wn.load_dict("bet_network.json")

"""
Explanation: save_dict and load_dict
These functions save and load Python dictionaries. This is convenient when you want to save crawl data.
End of explanation
"""


# Just in case we don't want to re-run the crawl, we will load the data directly
import wikinetworking as wn
import networkx as nx
%matplotlib inline
bet_undirected_graph_data = wn.load_dict("bet_network.json")

# Now create the graph
graph = wn.create_graph(bet_undirected_graph_data)
layout = nx.spring_layout(graph)

"""
Explanation: create_graph
Now that we have a dictionary representing an undirected graph, we need to turn it into a networkx.graph object which can be drawn. Optionally, we can pre-generate a node layout and save it.
End of explanation
"""


graph_html = wn.make_interactive_graph(graph, pos=layout)

"""
Explanation: make_interactive_graph
The networkx.graph object can be rendered as an interactive, clickable graph.
End of explanation
"""


with open("bet_network.html", "w") as f:
    f.write(graph_html)
    f.close()

"""
Explanation: You can save the resulting HTML of the interactive graph to a file, if you wish to load it outside of this notebook.
End of explanation
"""


wn.save_big_graph(graph, pos=layout)

"""
Explanation: save_big_graph
If you would like to save a very high resolution version of this graph for use on a display system like Texas Advanced Computing Center's Stallion, you can use the save_big_graph function. The default output size at 600dpi will be 4800x3600 pixels. Warning: This function can take a little while to run.
End of explanation
"""


