What roadblocks are there to HSA becoming standard, similar to floating point units becoming standard?
A couple issues come to mind:
Synchronization/Communication overhead
In order to seamlessly transition from CPU to GPU code you need to communicate with the GPU. The GPU additionally has to be available(aka not rendering the screen), and all instructions on the CPU side of things need to retire/finish executing. Additionally you need to make sure that any pending writes have reached L3 cache/main memory, so that the GPU sees writes. As a result a transition to GPU code is quiet expensive, especially if the GPU is doing something latency sensitive(like rendering the next frame of something), and you need to wait for that process/task/thread/whatever to finish. Similarly returning back to the CPU is also expensive.
In addition you have to handle what happens if multiple CPU cores start fighting over the GPU.
Differing Memory Performance Needs
GPUs typically require high bandwith memory, but low latency is not as important, while CPUs are typically more sensitive to low latency.
Low performance GPUs can and do use main memory, but if you wanted a high performance GPU built into the CPU you would potentially need two different types of memory. At which point there isn't much advantage to having everything on one chip, since all that does is make cooling harder.
Inertia/Dev Infrastructure
SIMD has compiler support right now and lots of work put into it. Simple GPU style workloads like dot products are already memory bound anyway on a CPU, so existing CPU+GPU combos would not benefit.
Could just have lots of SIMD
Not much more to say beyond heading. SIMD+Many cores+lots of execution units would give you a more GPU like CPU. Add better SMT for a bonus. See Xeon Phi for a real world implementation of this concept. Though one thing worth mentioning is silicon spent on more GPU style features is silicon not spent on branch prediction etc.
Edit:
Another thing that comes to mind is there are broadly speaking three reasons to have a GPU.

Just want to browse the web, display Netflix etc. For this use case existing CPU and GPU performance/architecture is more than sufficient.
Want to play high end videogames etc. Existing architecture has a lot of momentum behind it, and I'm not convinced gaming CPU workloads really need better SIMD performance, and instead need better cache/branch etc, though I don't really know. However the GPU is likely already busy so it might not be the best idea to shift even more work to the CPU
HPC applications. Custom hardware like Xeon Phi is available for people who need a more GPU like CPU.