Why do HSK lists differ from character frequency lists?
To answer your second question:

From a mathematical perspective, it will depend on the texts that are sampled to compile any list and the variances associated with those samples. Languages evolve and there is no true character frequency, only an approximation. Spoken Chinese will have a different character frequency to written Chinese because the spoken language has a high entropy. It is likely that the HSK test had a panel of experts select characters based on what they deemed to be suitable, whereas the list you referenced was mass sampled by some kind of web crawler.

It is inconceivable that every single electronic Chinese character on the internet has been sampled, and if it was that would be very biased as that would ignore all of the written works not uploaded to the internet. Depending on the format of the inputs there will be a huge difference, if lots of text was taken from 微薄, then the character 哈 might rank as common, purely from comments that are like "hahahaha" for example. If classical Chinese was sampled heavily then characters like 曰 might feature much more frequently than 说.

Some of the HSK 5 and 6 words are very political and business oriented, but there are not many scientific words.