Title: Scrapy couldn't parse some html file correctly
Tags: python;scrapy
Question: I have used Scrapy a few weeks and recently, I have found HtmlXPathSelector couldn't parse some html file properly.
In the web page http://detail.zol.com.cn/series/268/10227_1.html , there's only a tag named
`div id='param-more' class='mod_param  '`.
When I used the xpath "//div[@id='param-more']" to select the tag, it returned [].
I have tried scrapy shell and got the same results.
When using wget to retrieve the web page, I could also find the tag "div id='param-more' class='mod_param  '" in the html source file and I think it's not caused by the reason that the tag is displayed by triggering an action.
Please give me some tips on how to solve this problem.
The following is the code sinppet about the problem. When processing the above url, len(nodes_product) is always 0
def parse_series(self, response):
hxs = HtmlXPathSelector(response)
xpath_product = "//div[@id='param-normal']/table//td[@class='name']/a | "\
"//div[@id='param-more']/table//td[@class='name']/a"
nodes_product = hxs.select(xpath_product)
if len(nodes_product) == 0:
# there's only the title, no other products in the series
.......
else:
.......
Comment: The code snippet was added just now :)

Comment: Please you show the specific code you used that returned the incorrect result.

Here is the accepted answer: This appears to be a bug with XPathSelectors. I created a quick test spider and ran into the same problem. I believe it has something to do with the non-standard characters on the page.
I do not believe the problem is that the 'param-more' div is associated with any javascript event or CSS hiding. I disabled javascript and also changed my user-agent (and location) to see if this effected the data on the page. It didn't.
I was, however, able to parse the 'param-more' div using beautifulsoup:
from scrapy.selector import HtmlXPathSelector
from scrapy.spider import BaseSpider
from bs4 import BeautifulSoup
class TestSpider(BaseSpider):
name = "Test"
start_urls = [
"http://detail.zol.com.cn/series/268/10227_1.html"
]
def parse(self, response):
hxs = HtmlXPathSelector(response)
#data = hxs.select("//div[@id='param-more']").extract()
data = response.body
soup = BeautifulSoup(data)
print soup.find(id='param-more')
Someone else may know more about the XPathSelect issue, but for the time being, you can save the HTML found by beautifulsoup to an item and pass it into the pipeline.
Here is the link to the most recent beautifulsoup version: http://www.crummy.com/software/BeautifulSoup/#Download
UPDATE
I believe I found the specific issue. The webpage being discussed specifies in a meta tag that it uses the GB 2312 charset. The conversion from GB 2312 to unicode is problematic because there are some characters which do not have a unicode equivalent. This would not be an issue, except for the fact that UnicodeDammit, beautifulsoup's encoding detection module, actually determines the encoding to be ISO 8859-2. The problem is that lxml determines the encoding of a document by looking at the charset specified in the meta tag of the header. Thus, there is an encoding type mismatch between what lxml and scrapy perceive.
The following code demonstrates the above problem, and provides an alternative to having to rely on the BS4 library:
from scrapy.selector import HtmlXPathSelector
from scrapy.spider import BaseSpider
from bs4 import BeautifulSoup
import chardet
class TestSpider(BaseSpider):
name = "Test"
start_urls = [
"http://detail.zol.com.cn/series/268/10227_1.html"
]
def parse(self, response):
encoding = chardet.detect(response.body)['encoding']
if encoding != 'utf-8':
response.body = response.body.decode(encoding, 'replace').encode('utf-8')
hxs = HtmlXPathSelector(response)
data = hxs.select("//div[@id='param-more']").extract()
#print encoding
print data
Here, you see that by forcing lxml to use utf-8 encoding, it does not attempt to map from what it perceives as GB 2312->utf-8.
In scrapy, the HTMLXPathSelectors encoding is set in the scrapy/select/lxmlsel.py module. This module passes the response body to the lxml parser using the response.encoding attribute, which is ultimately set in the scrapy/http/response/test.py module.
The code that handles setting the response.encoding attribute is as follows:
@property
def encoding(self):
return self._get_encoding(infer=True)
def _get_encoding(self, infer=False):
enc = self._declared_encoding()
if enc and not encoding_exists(enc):
enc = None
if not enc and infer:
enc = self._body_inferred_encoding()
if not enc:
enc = self._DEFAULT_ENCODING
return resolve_encoding(enc)
def _declared_encoding(self):
return self._encoding or self._headers_encoding() \
or self._body_declared_encoding()
The important thing to note here is that _headers_encoding and _encoding both will ultimately reflect the encoding declared in the meta tag in the header over actually using something like UnicodeDammit or chardet to determine the documents encoding. Thus, situations will arise where a document contains invalid characters for the encoding it has specified it has, and I believe that Scrapy will overlook this, ultimately resulting in the problem we are seeing today.
Comment for this answer: The **HtmlXPathSelector** use **lxml** as backend by default and maybe it has something wrong with the html source file that **lxml** could not handle. I will use **BeautifulSoup** to handle such web pages when the problem arises. Thank you very much :D
Comment for this answer: Wow, it works! However, I crawled thounds of simliar pages like [http://detail.zol.com.cn/series/268/10227_1.html] and only this page had the problem. For example, [http://detail.zol.com.cn/series/16/14042_1.html] has a simliar tag "div id='param-normal''" and I can get the tag using HtmlXPathSelector. These pages have the same 'charset'. Checking the  charset before handling the html source file is a good practice. Thanks again :D
Comment for this answer: *Found the problem specifically. I provided a alternative to my previous method of getting around the issue.
Comment for this answer: I think the issue was though that on the original page, there were illegal characters that did not map correctly to unicode because they were of a differetn charset than that specified in the meta tag. My guess is, that if it worked on other pages with similar structure and content, then only the first page had the illegal incorrectly-mapping characters. They were probably outside the range of those recognized by the charmap.
Anyway, I am glad that you got it to work! Congrats!

Here is another answer: 'mod_param ' != 'mod_param'
The class does not equal "mod_param" but it does contain "mod_param", note there is a blank space on the end:
stav@maia:~$ scrapy shell http://detail.zol.com.cn/series/268/10227_1.html
2012-08-23 09:17:28-0500 [scrapy] INFO: Scrapy 0.15.1 started (bot: scrapybot)
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
IPython 0.12.1 -- An enhanced Interactive Python.
In [1]: hxs.select("//div[@class='mod_param']")
Out[1]: []
In [2]: hxs.select("//div[contains(@class,'mod_param')]")
Out[2]: [<HtmlXPathSelector xpath="//div[contains(@class,'mod_param')]" data=u'<div id="param-more" class="mod_param  "'&gt;]
In [3]: len(hxs.select("//div[contains(@class,'mod_param')]").extract())
Out[3]: 1
In [4]: len(hxs.select("//div[contains(@class,'mod_param')]").extract()[0])
Out[4]: 5372
Comment for this answer: Thank you for your help in scrapy-user maillist :) But when I tried your method, it still got the same results **[]**

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 2, 'CommentCount': 2, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2012:00:08/23/12', 'FavoriteCount': 1.0, 'PostTypeId': 1, 'Tags': '<python><scrapy>', 'Title': "Scrapy couldn't parse some html file correctly", 'ViewCount': 2262, 'id': 12084033, 'user_AboutMe': '', 'user_AccountId': 473640.0, 'user_DisplayName': 'flyer', 'user_Reputation': 8724.0, 'user_Views': 369.0, 'user_id': 882862.0}}