Philip Garvey, 56, his wife Ann, 55, and their children Emily, 23, and Daniel, 20, died when their plane came down in Somerset in November 2015.
They were flying from Surrey to Devon.
The Air Accidents Investigation Branch (AAIB) said it was possible Mr Garvey tried to manually start landing while the autopilot was still on.
Mr Garvey had taken off from Fairoakes Airport, close to the family home in Woking, and was heading to Dunkeswell Aerodrome.
The AAIB's report said it was an overcast and rainy day and all planned flights at Dunkeswell had been cancelled because of the weather.
Several witnesses told investigators they heard a sound like a plane doing "aerobatics" before the Piper Malibu Mirage appeared from the clouds and hit the ground near Churchinford, south of Taunton.
The AAIB said: "The evidence from the autopilot examination system suggested that, as the pilot turned on to the final approach and started to descend, the autopilot may not have been disengaged due to a mental lapse, incorrect button selection or a technical fault."
The plane was seen to quickly climb into cloud before coming down.
The AAIB concluded it was "unable to determine with certainty the reason for the initial rapid climb" but said attempting to override the autopilot would have caused the plane to go "nose up" as it would fight to maintain its set height.
While the aircraft could still have been controllable, it said "recovery may have been beyond his capabilities".
Mr Garvey had been a pilot since 2012 and had flown to Dunkeswell 25 times since 2013, including 14 times from Fairoaks.
His instructors told investigators he was "less able than average pilots at multi-tasking and poor at prioritising especially when under pressure".
Another instructor said Mr Garvey "would often have difficulty dealing with an unusual or unexpected condition".
A pilot who died along with his family when the plane he was flying crashed was "poor under pressure", his instructors have told investigators.