Scraping webpage elements using Python
My take, in short:

in programming there is nothing worse than silently swallowing exceptions :) And the fact that you are catching any exception, and not just those related to those specific operations, means than all kinds of errors will go unnoticed.
At the very least, the exceptions should be logged somewhere
be more specific and catch relevant exceptions instead, usually it will be stuff like HTMLParser.HTMLParseError and also requests.exceptions - experiment a bit
Note the difference in behavior between find and find_all since you are using both functions:


If find_all() can’t find anything, it returns an empty list. If find()
can’t find anything, it returns None


Armed with that knowledge you can simply test your expressions for None before you try to fetch the value with find - it's better to avoid an exception than having to handle it.
Likewise, if bs4.find_all('a', {'class': 'question-hyperlink'}) returns an empty list, you don't proceed with the list comprehension but you assign some default value, possibly: payload['hyperlinks'] = None

So the key here is to test the result of every call to find or find_all and act accordingly. Then you can get rid of those try catch blocks. Only one all-purpose exception handler for the whole procedure shall suffice.