<p>What's the best way to go about validating that a document follows some version of HTML (prefereably that I can specify)? I'd like to be able to know where the failures occur, as in a web-based validator, except in a native Python app.</p>

[{'answer_id': 35543, 'author': 'John Millikin', 'author_id': 3560, 'author_profile': 'https://Stackoverflow.com/users/3560', 'pm_score': 5, 'selected': True, 'text': '<p>XHTML is easy, use <a href="http://lxml.de/validation.html" rel="noreferrer">lxml</a>.</p>\n\n<pre><code>from lxml import etree\nfrom StringIO import StringIO\netree.parse(StringIO(html), etree.HTMLParser(recover=False))\n</code></pre>\n\n<p>HTML is harder, since there\'s traditionally not been as much interest in validation among the HTML crowd (run StackOverflow itself through a validator, yikes). The easiest solution would be to execute external applications such as <a href="http://www.jclark.com/sp/" rel="noreferrer">nsgmls</a> or <a href="http://openjade.sourceforge.net/" rel="noreferrer">OpenJade</a>, and then parse their output.</p>\n'}, {'answer_id': 35562, 'author': 'Neall', 'author_id': 619, 'author_profile': 'https://Stackoverflow.com/users/619', 'pm_score': 2, 'selected': False, 'text': '<p>I think that <a href="http://tidy.sourceforge.net/" rel="nofollow noreferrer">HTML tidy</a> will do what you want. There is a Python binding for it.</p>\n'}, {'answer_id': 35572, 'author': 'Aaron Maenpaa', 'author_id': 2603, 'author_profile': 'https://Stackoverflow.com/users/2603', 'pm_score': 3, 'selected': False, 'text': '<p>Try tidylib. You can get some really basic bindings as part of the elementtidy module (builds elementtrees from HTML documents). <a href="http://effbot.org/downloads/#elementtidy" rel="noreferrer">http://effbot.org/downloads/#elementtidy</a></p>\n\n<pre><code>&gt;&gt;&gt; import _elementtidy\n&gt;&gt;&gt; xhtml, log = _elementtidy.fixup("&lt;html&gt;&lt;/html&gt;")\n&gt;&gt;&gt; print log\nline 1 column 1 - Warning: missing &lt;!DOCTYPE&gt; declaration\nline 1 column 7 - Warning: discarding unexpected &lt;/html&gt;\nline 1 column 14 - Warning: inserting missing \'title\' element\n</code></pre>\n\n<p>Parsing the log should give you pretty much everything you need.</p>\n'}, {'answer_id': 646877, 'author': 'karlcow', 'author_id': 62262, 'author_profile': 'https://Stackoverflow.com/users/62262', 'pm_score': 4, 'selected': False, 'text': '<p>You can decide to install the HTML validator locally and create a client to request the validation. </p>\n\n<p>Here I had made a program to validate a list of urls in a txt file. I was just checking the HEAD to get the validation status, but if you do a GET you would get the full results. Look at the API of the validator, there are plenty of options for it.</p>\n\n<pre><code>import httplib2\nimport time\n\nh = httplib2.Http(".cache")\n\nf = open("urllistfile.txt", "r")\nurllist = f.readlines()\nf.close()\n\nfor url in urllist:\n   # wait 10 seconds before the next request - be nice with the validator\n   time.sleep(10)\n   resp= {}\n   url = url.strip()\n   urlrequest = "http://qa-dev.w3.org/wmvs/HEAD/check?doctype=HTML5&amp;uri="+url\n   try:\n      resp, content = h.request(urlrequest, "HEAD")\n      if resp[\'x-w3c-validator-status\'] == "Abort":\n         print url, "FAIL"\n      else:\n         print url, resp[\'x-w3c-validator-status\'], resp[\'x-w3c-validator-errors\'], resp[\'x-w3c-validator-warnings\']\n   except:\n      pass\n</code></pre>\n'}, {'answer_id': 1279293, 'author': 'Dave Brondsema', 'author_id': 79697, 'author_profile': 'https://Stackoverflow.com/users/79697', 'pm_score': 5, 'selected': False, 'text': '<p><a href="http://countergram.github.io/pytidylib/" rel="noreferrer">PyTidyLib</a> is a nice python binding for HTML Tidy. Their example:</p>\n\n<pre><code>from tidylib import tidy_document\ndocument, errors = tidy_document(\'\'\'&lt;p&gt;f&amp;otilde;o &lt;img src="bar.jpg"&gt;\'\'\',\n    options={\'numeric-entities\':1})\nprint document\nprint errors\n</code></pre>\n\n<p>Moreover it\'s compatible with both <a href="http://tidy.sourceforge.net/" rel="noreferrer">legacy HTML Tidy</a> and the <a href="http://www.html-tidy.org/" rel="noreferrer">new tidy-html5</a>.</p>\n'}, {'answer_id': 10519634, 'author': 'Martin Hepp', 'author_id': 516699, 'author_profile': 'https://Stackoverflow.com/users/516699', 'pm_score': 4, 'selected': False, 'text': '<p>I think the most elegant way it to invoke the W3C Validation Service at</p>\n\n<pre><code>http://validator.w3.org/\n</code></pre>\n\n<p>programmatically. Few people know that you do not have to screen-scrape the results in order to get the results, because the service returns non-standard HTTP header paramaters </p>\n\n<pre><code>X-W3C-Validator-Recursion: 1\nX-W3C-Validator-Status: Invalid (or Valid)\nX-W3C-Validator-Errors: 6\nX-W3C-Validator-Warnings: 0\n</code></pre>\n\n<p>for indicating the validity and the number of errors and warnings.</p>\n\n<p>For instance, the command line </p>\n\n<pre><code>curl -I "http://validator.w3.org/check?uri=http%3A%2F%2Fwww.stalsoft.com"\n</code></pre>\n\n<p>returns</p>\n\n<pre><code>HTTP/1.1 200 OK\nDate: Wed, 09 May 2012 15:23:58 GMT\nServer: Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2\nContent-Language: en\nX-W3C-Validator-Recursion: 1\nX-W3C-Validator-Status: Invalid\nX-W3C-Validator-Errors: 6\nX-W3C-Validator-Warnings: 0\nContent-Type: text/html; charset=UTF-8\nVary: Accept-Encoding\nConnection: close\n</code></pre>\n\n<p>Thus, you can elegantly invoke the W3C Validation Service and extract the results from the HTTP header:</p>\n\n<pre class="lang-python prettyprint-override"><code># Programmatic XHTML Validations in Python\n# Martin Hepp and Alex Stolz\n# mhepp@computer.org / alex.stolz@ebusiness-unibw.org\n\nimport urllib\nimport urllib2\n\nURL = "http://validator.w3.org/check?uri=%s"\nSITE_URL = "http://www.heppnetz.de"\n\n# pattern for HEAD request taken from \n# http://stackoverflow.com/questions/4421170/python-head-request-with-urllib2\n\nrequest = urllib2.Request(URL % urllib.quote(SITE_URL))\nrequest.get_method = lambda : \'HEAD\'\nresponse = urllib2.urlopen(request)\n\nvalid = response.info().getheader(\'X-W3C-Validator-Status\')\nif valid == "Valid":\n    valid = True\nelse:\n    valid = False\nerrors = int(response.info().getheader(\'X-W3C-Validator-Errors\'))\nwarnings = int(response.info().getheader(\'X-W3C-Validator-Warnings\'))\n\nprint "Valid markup: %s (Errors: %i, Warnings: %i) " % (valid, errors, warnings)\n</code></pre>\n'}, {'answer_id': 39336630, 'author': 'user9869932', 'author_id': 1183098, 'author_profile': 'https://Stackoverflow.com/users/1183098', 'pm_score': 2, 'selected': False, 'text': '<p>In my case the python W3C/HTML cli validation packages did not work (as of sept 2016).</p>\n<p>I did it manually using <code>requests</code> like so</p>\n<p>code:</p>\n<pre><code>r = requests.post(\'https://validator.w3.org/nu/\', \n    data=open(\'FILE.html\',\'rb\').read(), params={\'out\': \'json\'}, \n    headers={\'User-Agent\': \'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101    Safari/537.36\', \n    \'Content-Type\': \'text/html; charset=UTF-8\'})\n\nprint r.json()\n</code></pre>\n<p>in the console:</p>\n<pre><code>$ echo \'&lt;!doctype html&gt;&lt;html lang=en&gt;&lt;head&gt;&lt;title&gt;blah&lt;/title&gt;&lt;/head&gt;&lt;body&gt;&lt;/body&gt;&lt;/html&gt;\' | tee FILE.html \n$ pip install requests\n\n$ python\nPython 2.7.12 (default, Jun 29 2016, 12:46:54)\n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin\nType &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.\n\n&gt;&gt;&gt; import requests\n\n&gt;&gt;&gt; r = requests.post(\'https://validator.w3.org/nu/\', \n...                    data=open(\'FILE.html\', \'rb\').read(), \n...                    params={\'out\': \'json\'}, \n...                    headers={\'User-Agent\': \'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36\', \n...                    \'Content-Type\': \'text/html; charset=UTF-8\'})\n\n&gt;&gt;&gt; r.text\n&gt;&gt;&gt; u\'{&quot;messages&quot;:[]}\\n\'\n\n&gt;&gt;&gt; r.json()\n&gt;&gt;&gt; {u\'messages\': []}\n</code></pre>\n<p>More documentation here <a href="http://docs.python-requests.org/en/master/user/quickstart/#binary-response-content" rel="nofollow noreferrer">python requests</a>, <a href="https://github.com/validator/validator/wiki/Service:-Input:-POST-body" rel="nofollow noreferrer">W3C Validator API</a></p>\n'}, {'answer_id': 40229004, 'author': 'speedplane', 'author_id': 234270, 'author_profile': 'https://Stackoverflow.com/users/234270', 'pm_score': 2, 'selected': False, 'text': '<p>This is a very basic html validator based on lxml\'s HTMLParser. It is not a complete html validator, but does a few basic checks, doesn\'t require any internet connection, and doesn\'t require a large library.</p>\n\n<pre><code>_html_parser = None\ndef validate_html(html):\n    global _html_parser\n    from lxml import etree\n    from StringIO import StringIO\n    if not _html_parser:\n        _html_parser = etree.HTMLParser(recover = False)\n    return etree.parse(StringIO(html), _html_parser)\n</code></pre>\n\n<p>Note that this will not check for closing tags, so for example, the following will pass:</p>\n\n<pre><code>validate_html("&lt;a href=\'example.com\'&gt;foo")\n&gt; &lt;lxml.etree._ElementTree at 0xb2fd888&gt;\n</code></pre>\n\n<p>However, the following wont:</p>\n\n<pre><code>validate_html("&lt;a href=\'example.com\'&gt;foo&lt;/a")\n&gt; XMLSyntaxError: End tag : expected \'&gt;\', line 1, column 29\n</code></pre>\n'}, {'answer_id': 60886392, 'author': 'Changaco', 'author_id': 2729778, 'author_profile': 'https://Stackoverflow.com/users/2729778', 'pm_score': 3, 'selected': False, 'text': '<p>The <a href="https://pypi.org/project/html5lib/" rel="noreferrer">html5lib</a> module can be used to validate an HTML5 document:</p>\n\n<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; import html5lib\n&gt;&gt;&gt; html5parser = html5lib.HTMLParser(strict=True)\n&gt;&gt;&gt; html5parser.parse(\'&lt;html&gt;&lt;/html&gt;\')\nTraceback (most recent call last):\n  ...\nhtml5lib.html5parser.ParseError: Unexpected start tag (html). Expected DOCTYPE.\n</code></pre>\n'}]
['https://Stackoverflow.com/questions/35538', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/3594/']