Want to pull a journal title from an RCSB Page using python & BeautifulSoup
Question: I am trying to get specific information about the original citing paper in the
Protein Data Bank given only the 4 letter PDBID of the protein.

To do this I am using the python libraries requests and BeautifulSoup. To try
and build the code, I went to the page for a particular protein, in this case
1K48, and also save the HTML for the page (by hitting command+s and saving the
HTML to my desktop).

First things to note:

1) The url for this page is:
<http://www.rcsb.org/pdb/explore.do?structureId=1K48>

2) You can get to the page for any protein by replacing the last four
characters with the appropriate PDBID.

3) I am going to want to perform this procedure on many PDBIDs, in order to
sort a large list by the Journal they originally appeared in.

4) Searching through the HTML, one finds the journal title located inside a
form here:

    
    
    <form action="http://www.rcsb.org/pdb/search/smartSubquery.do" method="post" name="queryForm">  
        <p><span id="se_abstractTitle"><a onclick="c(0);">Refined</a> <a onclick="c(1);">structure</a> <a onclick="c(2);">and</a> <a onclick="c(3);">metal</a> <a onclick="c(4);">binding</a> <a onclick="c(5);">site</a> of the <a onclick="c(8);">kalata</a> <a onclick="c(9);">B1</a> <a onclick="c(10);">peptide.</a></span></p>                                                        
        <p><a class="sePrimarycitations se_searchLink" onclick="searchCitationAuthor(&#39;Skjeldal, L.&#39;);">Skjeldal, L.</a>,&nbsp;&nbsp;<a class="sePrimarycitations se_searchLink" onclick="searchCitationAuthor(&#39;Gran, L.&#39;);">Gran, L.</a>,&nbsp;&nbsp;<a class="sePrimarycitations se_searchLink" onclick="searchCitationAuthor(&#39;Sletten, K.&#39;);">Sletten, K.</a>,&nbsp;&nbsp;<a class="sePrimarycitations se_searchLink" onclick="searchCitationAuthor(&#39;Volkman, B.F.&#39;);">Volkman, B.F.</a></p> 
        <p>
            <b>Journal:</b>     
            (2002)
            <span class="se_journal">Arch.Biochem.Biophys.</span>
            <span class="se_journal"><b>399: </b>142-148</span>         
        </p>
    

A lot more is in the form but it is not relevant. What I do know is that my
journal title, "Arch.Biochem.Biophys", is located within a span tag with class
"se_journal".

And so I wrote the following code:

    
    
    def JournalLookup():
        PDBID= '1K48'
    
        import requests
        from bs4 import BeautifulSoup
    
        session = requests.session()
    
        req = session.get('http://www.rcsb.org/pdb/explore.do?structureId=%s' %PDBID)
    
        doc = BeautifulSoup(req.content)
        Journal = doc.findAll('span', class_="se_journal")
    

Ideally I'd be able to use find instead of findAll as these are the only two
in the document, but I used findAll to at least verify I'm getting an empty
list. I assumed that it would return a list containing the two span tags with
class "se_journal", but it instead returns an empty list.

After spending several hours going through possible solutions, including a
piece of code that printed every span in doc, I have concluded that the
requests doc does not include the lines I want at all.

Does anybody know why this is the case, and what I could possibly do to fix
it?

Thanks.


Answer: The content you are interested in is provided by the javascript. It's easy to
find out, visit the same URL on browser with javascript disabled and you will
not see that specific info. It also displays a friendly message:

> "This browser is either not Javascript enabled or has it turned off. This
> site will not function correctly without Javascript."

For javascript driven pages, you cannot use Python Requests. There are some
alternatives, one being [dryscape](https://github.com/niklasb/dryscrape).

PS: Do not import libraries/modules within a function. Python does not
recommend it and [PEP08](http://www.python.org/dev/peps/pep-0008/) says that:

> Imports are always put at the top of the file, just after any module
> comments and docstrings, and before module globals and constants.

[This SO question](http://stackoverflow.com/questions/128478/should-python-
import-statements-always-be-at-the-top-of-a-module) explains why it's not
recomended way to do it.

