Drift in PID controller
Your controller controls pitch and roll. However, in the graph you showed us we only see x,y and z.
Could you add the plot for pitch and roll? I'm not an expert in aircraft control, but I think you can have a constant pitch and roll and still move in the x,y,z plane.
I'll edit my answer as soon as you give more info.
Edit : You roll seems stable, you could try adding an integral component to your pitch roll controller.