How to remove the default cube?
Delete the cube:
Starting with the unchanged default scene, select the cube (RMB click on the cube) then hit X or DELETE and RMB click the following popup. You may also repeat this step on the camera and light if you wish to start with a completely blank scene.

Save the startup file:
Click File > Save Startup File or hit CtrlU and click to confirm. The scene is now saved as the default startup scene and you no longer have the default cube!