Getting last ten codepad.org posts with Easy Html Parser (EHP) python
Question: I have found a python html parser that builds a dom like structure for html
sources it seems easy to use and very fast. i'm trying to write a scraper for
codepad.org that retrieves the last ten posts from <http://codepad.org/recent>

The EHP lib is at <https://github.com/iogf/ehp> i have this code below that is
working.

    
    
    import requests
    from ehp import Html
    
    def catch_refs(data):
        html = Html()
        dom = html.feed(data)
    
        return [ind.attr['href']
                    for ind in dom.find('a')
                        if 'view' in ind.text()]
    
    def retrieve_source(refs, dir):
        """
        Get the source code of the posts then save in a dir.
        """
        pass
    
    
    if __name__ == '__main__':
        req  = requests.get('http://codepad.org/recent')
        refs = catch_refs(req.text)
        retrieve_source(refs, '/tmp/')
        print refs    
    

it outputs:

    
    
    [u'http://codepad.org/aQGNiQ6t', 
     u'http://codepad.org/HMrG1q7t', 
     u'http://codepad.org/zGBMaKoZ', ...]
    

as expected but i cant figure out how to download the source code of the
files.


Answer: Actually your `retrieve_source(refs, dir)` don't do anything.

So you are not getting any result.

Update according to your comment:

    
    
    import os
    
    
    def get_code_snippet(page):
        dom = Html().feed(page)
        # getting all <div class=='highlight'>
        elements = [el for el in dom.find('div')
                    if el.attr['class'] == 'highlight']
        return elements[1].text()
    
    def retrieve_source(refs, dir):
        for i, ref in enumerate(refs):
            with open(os.path.join(dir, str(i) + '.html'), 'w') as r:
                r.write(get_code_snippet(requests.get(ref).content))
    

