ICSP / ISP issue when programming ATmega328P
Sorry to say that you are right the problem is with your pulled-down resistors on programming pins.
As see in the picture there are mainly 6 pins required for programming Vcc, Gnd, RST, SCK, MISO, MOSI.

I have faced a similar kind of problem once I have designed a PCB based on ATmega328p and its job was to control few stepper motors and for homing the motor I have used limit switches and back then I have added one of the limits switches to PB4 (MISO) pin of MCU and was using internal pullup for the switch.
So all have tested my PCB all was working well and once while I was programming the limit switch on MISO pin was pressed hence connected to ground then no matter how hard I try to program It was not even detecting my MCU. was using a UABASP to program after 6 hr of debugging I accidentally noted that its due to a switch and after releasing switch to open state all worked well.
So you can try removing the resistor first from MISO and if it worked then great else get ready to remove 2 more resistors and your problem should be fixed after removing 3 resistors.
Also, it's good to have an onboard pull-up resistor on the reset pin.
Last I can see there is a difference in connection that you provided to your ICSP connector than the standards as in the image so hope you have done that intentionally.