Arduino controlled power supply
Update 24-March-2020. This circuit only works by luck. There is a better solution in my follow up question: Why does a bipolar transistor not work, but a MOSFET does

After much considered work (a.k.a. randomly trying stuff until it works), I've worked out that if I replace the transistor with a MOSFET I can achieve the desired result.

Following is the modified circuit. FWIW, the diode in the question diagram is incorrect, it is actually a 1N914 that I am using. So although the first diagram is not strictly correct, the only component I have changed is the transistor for the MOSFET.

Also, FWIW, I've triple checked the connection for the MOSFET.

Pin 1 (Gate) is connected to R1
Pin 2 (Drain) is connected to VCC
Pin 3 (Source) is connected to VIN (Arduino)

I would have expected that Pin 2 and 3 should be reversed, but if I try that (i.e. 2->VIN and 3->VCC) then the Arduino is permanently on. I will post a separate question asking about why this circuit works but the transistor one does not.