crawling web data using python html error
Question: i want to crawling data using python i tried tried again but it didn't work i
can not found code's error i wrote code like this:

    
    
    import re
    import requests
    from bs4 import BeautifulSoup
    
    url='http://news.naver.com/main/ranking/read.nhn?mid=etc&sid1=111&rankingType=popular_week&oid=277&aid=0003773756&date=20160622&type=1&rankingSectionId=102&rankingSeq=1'
    html=requests.get(url)
    #print(html.text)
    a=html.text
    bs=BeautifulSoup(a,'html.parser')
    print(bs)
    print(bs.find('span',attrs={"class" : "u_cbox_contents"}))
    

i want to crawl reply data in news

[![enter image description
here](http://i.stack.imgur.com/9hzGp.png)](http://i.stack.imgur.com/9hzGp.png)

as you can see, i tried to searing this:

span, class="u_cbox_contents" in bs

but python only say "None"

> None

so i check bs using function print(bs)

and i check up bs variable's contents

but there is no span, class="u_cbox_contents"

why this happing?

i really don't know why

please help me

thanks for reading.


Answer: Requests will fetch the URL's contents, but will not execute any JavaScript.

I performed the same fetch with cURL, and I can't find any occurrence of
`u_cbox_contents` in the HTML code. Most likely, it's injected using
JavaScript, which explains why BeautifulSoup can't find it.

If you need the page's code as it would be rendered in a "normal" browser, you
could try [Selenium](http://docs.seleniumhq.org). Also have a look at
[this](http://stackoverflow.com/questions/1916711/programmatic-python-browser-
with-javascript) SO question.

