Scan a webpage to find the start time and date for an event I am working on a simple web crawler that returns the start time and date for an event listed on a webpage. The webpage can be in two different formats and there are multiple other dates listed on the page. The part of the webpage I am scanning looks like this:

...
&lt;h4&gt;Time&lt;/h4&gt;
    &lt;p&gt; 
        Starts: &lt;time itemprop="startDate" datetime="2017-03-13T17:30:00ADT"&gt;Monday March 13, 2017 - 05:30 PM&lt;/time&gt;
        &lt;br /&gt;
        Ends: &lt;time itemprop="endDate" datetime="2017-03-13T19:00:00ADT"&gt;Monday March 13, 2017 - 07:00 PM&lt;/time&gt;
    &lt;/p&gt;

&lt;h4&gt;Additional Dates/Times&lt;/h4&gt;
    &lt;p&gt;
        Starts: &lt;time&gt;Monday January 30, 2017 - 05:30 PM&lt;/time&gt;
        &lt;br /&gt;
        Ends: &lt;time&gt;Monday January 30, 2017 - 07:00 PM&lt;/time&gt;
    &lt;/p&gt;&lt;div class="eventitemrepetition eventItemRepetition"&gt;&lt;p&gt;

&lt;p&gt;&lt;h4&gt;Location&lt;/h4&gt;
...


or like this:

...
&lt;h4&gt;Time&lt;/h4&gt;
&lt;p&gt;
    &lt;time itemprop="startDate" datetime="2017-03-17T13:30:00ADT"&gt;Friday March 17, 2017 - 01:30 PM&lt;/time&gt;
&lt;/p&gt;

&lt;p&gt;&lt;h4&gt;Location&lt;/h4&gt;
....


After using BeautifulSoup to find the links I am interested in I pass the link to these methods which find the text between 'Time' and 'Location'. Next it searches for a tag in that code block '">" which I know only appears before the date text and scans until the next closing tag, returning a date string.

MAX_CHARS = 140

def get_date(link):
    date_text = ""
    event_html = urllib2.urlopen(link['href']).read()
    start = find_date_location(event_html)
    # after the datetime closing tag the date begins until another tag opens
    for x in range(start, start + MAX_CHARS):
        if event_html[x] == '&lt;':
            break
        date_text += event_html[x]

    return date_text


def find_date_location(html):
    date_starts = html.find('&lt;h4&gt;Time&lt;/h4&gt;')
    date_ends = html.find('&lt;h4&gt;Location&lt;/h4&gt;')
    for x in range(date_starts, date_ends):
        if html[x] + html[x+1] == '"&gt;':
            return x+2

    raise ValueError('Date not found in HTML within time range')


This would return "Monday March 13, 2017 - 05:30 PM" in the first case and "Friday March 17, 2017 - 01:30 PM" in the second case. My method seems really hacky. Any tips on how I could do this better?
Your current approach is very fragile and might easily break if the markup changes even slightly - imagine, for example, opening and closing h4 being on separate lines while still being a valid HTML element.

I would use a proper HTML parser like BeautifulSoup instead (you mentioned you've already tried it). In order to locate the start and end dates, we may use the itemprop attribute:

from bs4 import BeautifulSoup


def get_even_date_range(html):
    soup = BeautifulSoup(html, 'html.parser')

    start_date = soup.find("time", itemprop="startDate")
    end_date = soup.find("time", itemprop="endDate")

    return (start_date.get_text() if start_date else None,
            end_date.get_text() if end_date else None)


Here, the get_even_date_range() function would return a tuple with start and end dates as items. It would return None if a date is not found. For the first sample input HTML, it would return:

('Monday March 13, 2017 - 05:30 PM', 'Monday March 13, 2017 - 07:00 PM')


And, for the second:

('Friday March 17, 2017 - 01:30 PM', None)




You can then go further and convert the date strings to datetimes using the datetime.strptime() and the %A %B %d, %Y - %H:%M %p format:

from datetime import datetime
from bs4 import BeautifulSoup


DATE_FORMAT = "%A %B %d, %Y - %H:%M %p"


def get_date(date_element):
    return datetime.strptime(date_element.get_text(), DATE_FORMAT) if date_element else None


def get_even_date_range(html):
    soup = BeautifulSoup(html, 'html.parser')

    start_date = soup.find("time", itemprop="startDate")
    end_date = soup.find("time", itemprop="endDate")

    return get_date(start_date), get_date(end_date)


Note that I've also moved the repetitive date retrieval logic to a separate reusable get_date() function.