I am trying to crawl the source of this website of a research article, exploiting a redirect from it's Document Object Identifier (DOI):doi=10.1006/jfin.1996.0208url=$(curl -L -s -w %{url_effective} http://dx.doi.org/$doi -o /dev/null)curl -L --referer ;auto $url > test.htm --dump-header /dev/nullThe file/source however looks different from what I see when I open the link in the browser. For example, the list of references is limited to 20 entries, and instead of showing links to these items, I only see Loading.Obviously, the source is different from what the browser (FF) shows me. Indeed, when I take a look at the site's source file using this bookmarklet to view generated source, I get the complete source file I want to have.Now is there a way to crawl the generated source of a webpage? Taking a look at this detailed manual, I couldn't find anything related to generated source.
Crawling generated source of page
curl