Altium slow from too many primitives
Sounds like it's possible the Online DRC might be screwing you up. Once you get to the stage of complex rule sets it bites itself in the ass and can seriously hold you up, especially if some things are being highlighted.

These complex rule sets can appear quite quickly in mixed signal or high-quality analogue designs (pour 28 mil from this, 5 mil from that, 5mil trace/clearance, except for analogue, because those need analogue ground pours around them, etc etc).

What happens on my pro set-up at the office is that turning off Online DRC will double the responsiveness if not quadruple about 60% into a design. 

Funnily enough, my home computer where I do my freelance work has this problem to a much smaller degree, so I am starting to formulate the idea that it might be the video card that it uses for some Online DRC viz. My home laptop has the Nvidea 970 with 6GB dedicated, whereas the office PC has Intel HD-something with shared RAM. (But more RAM in itself, 32G vs 16G)