Python Mechanize for form with options populated by Javascript
Question: I'm trying to use mechanize to grab prices for New York's metro-north railroad
from this site: <http://as0.mta.info/mnr/fares/choosestation.cfm>

The problem is that when you select the first option, the site uses javascript
to populate your list of possible destinations. I have written equivalent code
in python, but I can't seem to get it all working. Here's what I have so far:

    
    
    import mechanize
    import cookielib
    from bs4 import BeautifulSoup
    
    br = mechanize.Browser()
    br.set_handle_robots(False)
    br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1)     Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
    
    br.open("http://as0.mta.info/mnr/fares/choosestation.cfm")
    
    br.select_form(name="form1")
    br.form.set_all_readonly(False)
    
    origin_control = br.form.find_control("orig_stat", type="select")
    origin_control_list = origin_control.items
    origin_control.value = [origin_control.items[0].name]
    
    destination_control_list = reFillList(0, origin_control_list)
    
    destination_control = br.form.find_control("dest_stat", type="select")
    destination_control.items = destination_control_list
    destination_control.value = [destination_control.items[0].name]
    
    response = br.submit()
    response_text = response.read()
    print response_text
    

I know I didn't give you code for the reFillList() method, because it's long,
but assume it correctly creates a list of mechanize.option objects. Python
doesn't complain about me about anything, but on submit I get the html for
this alert:

"Fare information for travel between two lines is not available on-line.
Please contact our Customer Information Center at 511 and ask to speak to a
representative for further information."

Am I missing something here? Thanks for all the help!


Answer: It can't really be done without trying to make sense of the crazy logic in
that function. I suggest a js solution or a full browser like selenium.

