Why does `RCC_GetClocksFreq` return different results than I expect?
I managed to fix my problem by adding the following near the beginning:

RCC_HSEConfig(RCC_HSE_ON);


I had not started the HSE so most of the configuration was propably just ignored as the core still worked.

As a sidenote also the HCLK, PCLK1 and PCLK2 divisions were incorrect (over the frequency limit.) I had not spent so much attention on those as I tried to find out why SYSCLK was not getting set correctly.