How to move/remove/join adjacent Blade Speed points in FCPX
Partial answer...

I still don't know how to remove a single blade speed point.  It is possible to remove all speed points for a clip by selecting the clip and choosing a new speed (eg Normal 100%) from the Retiming menu.

I still don't know how to join adjacent segments (kind of same thing as above).

Found out how to move the blade speed point relative to the clip itself without retiming:


double-click the blade speed centre handle and click the 'Edit' button





You get this icon you can drag in the timeline:




(classic bit of Apple "hidden UI" gotcha)

If you are zoomed in and you have a big speed differential either side of the centre point then it may not be at all obvious that you really are moving the src frame (fooled me before) because the preview changes are complex. But with Normal speed both sides it is clear this does what I wanted.