Why can't I change/delete the world background image?
You can adjust the material preview world shading by clicking the drop-down to the right of the shading icons.

Enable 'Scene Lights' and 'Scene World' to use your scene's illumination instead of the material preview presets. You can select different HDRI worlds by clicking the sphere or adjust the other settings. Clicking the 'cog' takes you to the preferences window to allow you to set up custom lighting with your own HDRIs.