programming atmega2560 by ISP on self-made PCB
can it still work with only one VCC pin and one GND pin 


No. The MCU will not work correctly like that. I have seen various different incorrect behaviours when this design mistake has been made, so I am not surprised that you are having problems. :-(

As other experienced members have already kindly explained, you also need decoupling capacitors close to the MCU (see below). The oscillator and its capacitors need to be close to the MCU too.

Atmel Application Note: AVR042: AVR Hardware Design Considerations (522kB PDF) is full of advice, shows you all the Vcc and Gnd connections being used, decoupling capacitors, discusses PCB layout and oscillator placement etc. I recommend that you follow its suggestions.


Edit: Here is a related question being asked about using all of the Vcc (Vdd) and Gnd (Vss) pins on PIC MCUs - the answers apply here too, for the same reasons given:

If a PIC MCU provides multiple Vdd/Vss should you provide power to them all?


Edit: Adding update to summarise some comments on this answer, for future reference: AVcc must also be connected, even if the ADC is unused. The AVR datasheets specify how closely the AVcc voltage must be to the Vcc voltage, and an unconnected AVcc violates that requirement.

For example, the ATmega2560 datasheet specifies that AVcc must be between Vcc-0.3V to Vcc+0.3V. So if AVcc = 0V (unconnected), that exceeds that +/-0.3V difference from any operational Vcc voltage.