Extracting and normalizing URLs in an HTML document
Some stuff you could perhaps do differently:

# your code
if url is None or ' ' in url or '&lt;' in url or '&gt;' in url:
    continue

# the alternative
if url is None or any(char in url for char in ' &lt;&gt;'):
    continue


Also, you can call the split method directly, without the if statement, as it will return a single item list with the full string inside if the character is not in the string:

# your code
if '?' in url:
    url = url.split('?')[0]
if '#' in url:
    url = url.split('#')[0]

# the alternative
for splitter in '?#':
    url = url.split(splitter, 1)[0]


Notice the micro-optimization of using the second argument of split, so that the string is only split at the first occurrence if there is more than one.