Headset wobble after tightening
It sounds as through you loosened the compression nut inside your carbon steer tube, which you should not have done.  That bolt (the 5mm inside) should be tightened once during installation and is not touched for a headset adjustment (or than to perhaps check it is still tight).  When you loosened that bolt, you made it so that the top cap will simply pull the compression nut up, rather than pull the fork and compress the headset bearings.  Tighten the interior (the 5 mm inside) compression nut in place, then restart your headset adjustment process and hope that you haven't damaged the headset by riding it improperly adjusted.



By comparison, a fork with a metal steer tube will generally use a star nut.


These are hammered into place and will not move up.  The metal wings cut into the tube and allow the top cap to compress the headset.  This setup is inappropriate for a carbon steertube since it damages the carbon.