Keep Refraction and Remove the background object
There are a few methods you could use to achieve this.

The most practical ones are:


Material with a light path node
Compositing


for a simple scene like yours, I'd recommend the material option as you just need to add 3 nodes to your material setup and you can see the result in the viewport.

In my case the sphere is the refractive object, and the cube is the background object.

Add a mix shader between your existing shader and the material output node (in the background object material that is).
add a light path node -> is transmission ray to the fac input of the mix shader node.

add a transparent shader with a white color in the first (white emission shader in the screenshot) and your existing shader in the second shader slot (red emission shader in the screenshot).