MicroSD interfering with ISP programming
Because the SD card is an SPI device it shares the same pins as the ISP interface. The only way to not have the SD card interfere with the ISP is to not have the SD card connected.

As you have seen, that can be achieved by removing the card.

It can also be achieved by adding a tri-state buffer between the MCU and the card so the MCU only connects the card to the SPI bus when it is ready to use it - under default operation the card is disconnected.