ATMEGA328 USBasp schematic
The official USBasp has no connection on pin 3, so whether it is grounded or left open circuit doesn't matter. 

Pins 4 and 6 go to TXD and RXD in the programmer. TXD has a 270 &ohm; resistor in series so it is safe to 'short' it to ground, but better to leave it open circuit. RXD can be grounded or left open, as the serial port is not used in normal operation. Some USBASP clones ground both pins.

If you keep the programming cable and connections short then you should only need one ground wire, so just leave pins 3, 4 and 6 disconnected.