Parse birth and death dates from Wikipedia?
Question: I'm trying to write a python program that can search wikipedia for the birth
and death dates for people.

For example, Albert Einstein was born: 14 March 1879; died: 18 April 1955.

I started with [Fetch a Wikipedia article with
Python](http://stackoverflow.com/questions/120061/fetch-a-wikipedia-article-
with-python)

    
    
    import urllib2
    opener = urllib2.build_opener()
    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
    infile = opener.open('http://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&rvsection=0&titles=Albert_Einstein&format=xml')
    page2 = infile.read()
    

This works as far as it goes. `page2` is the xml representation of the section
from Albert Einstein's wikipedia page.

And I looked at this tutorial, now that I have the page in xml format...
<http://www.travisglines.com/web-coding/python-xml-parser-tutorial>, but I
don't understand how to get the information I want (birth and death dates) out
of the xml. I feel like I must be close, and yet, I have no idea how to
proceed from here.

**EDIT**

After a few responses, I've installed BeautifulSoup. I'm now at the stage
where I can print:

    
    
    import BeautifulSoup as BS
    soup = BS.BeautifulSoup(page2)
    print soup.getText()
    {{Infobox scientist
    | name        = Albert Einstein
    | image       = Einstein 1921 portrait2.jpg
    | caption     = Albert Einstein in 1921
    | birth_date  = {{Birth date|df=yes|1879|3|14}}
    | birth_place = [[Ulm]], [[Kingdom of Württemberg]], [[German Empire]]
    | death_date  = {{Death date and age|df=yes|1955|4|18|1879|3|14}}
    | death_place = [[Princeton, New Jersey|Princeton]], New Jersey, United States
    | spouse      = [[Mileva Marić]]&amp;nbsp;(1903–1919)&lt;br&gt;{{nowrap|[[Elsa Löwenthal]]&amp;nbsp;(1919–1936)}}
    | residence   = Germany, Italy, Switzerland, Austria, Belgium, United Kingdom, United States
    | citizenship = {{Plainlist|
    * [[Kingdom of Württemberg|Württemberg/Germany]] (1879–1896)
    * [[Statelessness|Stateless]] (1896–1901)
    * [[Switzerland]] (1901–1955)
    * [[Austria–Hungary|Austria]] (1911–1912)
    * [[German Empire|Germany]] (1914–1933)
    * United States (1940–1955)
    }}
    

So, much closer, but I still don't know how to return the death_date in this
format. Unless I start parsing things with `re`? I can do that, but I feel
like I'd be using the wrong tool for this job.


Answer: You can consider using a library such as
[BeautifulSoup](http://www.crummy.com/software/BeautifulSoup/bs4/doc/) or
[lxml](http://lxml.de) to parse the response html/xml.

You may also want to take a look at [`Requests`](http://docs.python-
requests.org/en/latest/index.html), which has a much cleaner API for making
requests.

* * *

Here is the working code using `Requests`, `BeautifulSoup` and `re`, arguably
not the best solution here, but it is quite flexible and can be extended for
similar problems:

    
    
    import re
    import requests
    from bs4 import BeautifulSoup
    
    url = 'http://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&rvsection=0&titles=Albert_Einstein&format=xml'
    
    res = requests.get(url)
    soup = BeautifulSoup(res.text, "xml")
    
    birth_re = re.search(r'(Birth date(.*?)}})', soup.revisions.getText())
    birth_data = birth_re.group(0).split('|')
    birth_year = birth_data[2]
    birth_month = birth_data[3]
    birth_day = birth_data[4]
    
    death_re = re.search(r'(Death date(.*?)}})', soup.revisions.getText())
    death_data = death_re.group(0).split('|')
    death_year = death_data[2]
    death_month = death_data[3]
    death_day = death_data[4]
    

* * *

Per @JBernardo's suggestion using JSON data and `mwparserfromhell`, a better
answer for this particular use case:

    
    
    import requests
    import mwparserfromhell
    
    url = 'http://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&rvsection=0&titles=Albert_Einstein&format=json'
    
    res = requests.get(url)
    text = res.json["query"]["pages"].values()[0]["revisions"][0]["*"]
    wiki = mwparserfromhell.parse(text)
    
    birth_data = wiki.filter_templates(matches="Birth date")[0]
    birth_year = birth_data.get(1).value
    birth_month = birth_data.get(2).value
    birth_day = birth_data.get(3).value
    
    death_data = wiki.filter_templates(matches="Death date")[0]
    death_year = death_data.get(1).value
    death_month = death_data.get(2).value
    death_day = death_data.get(3).value
    

