Bike seat tube is messed up
What’s happened here is that you have expanded the diameter of the top of the seat tube, in the process it looks like you have made some small cracks on the top edge, and created a number of burrs. That’s why the original seat post clamp will not fit anymore.
The larger clamp obviously does not work as its diameter is too large to clamp down on the tube with enough force.
Given that there is a reasonable length of seat tube above the top tube, I’d say that this is rescuable by carefully cutting the deformed area away. Removing 5mm from the top of the tube and smoothing the area below to remove burrs may be enough. The slot in the back of the tube may also need to be extended to allow the seat clamp to compress the tube.
No offense meant, but given that you’ve made a big mess of the seat tube with your own efforts, I’d enlist the help of experienced bike mechanic, who will also be able to assess whether the seat tube can be cut back safely.