Remove infinity wallpaper only from AOD
Use an external wallpaper app like Google's wallpaper app. Set the home and lock screen with a wallpaper. Then set the Infinity wallpaper you want, only for the home screen.
This worked for me.