Two tries from Mark Atkinson and one from Matt Scott helped put them 21-6 ahead, before Simon Labouyrie and Dion Evarard Oulai crossed for the hosts.
Billy Twelvetrees' try made it 29-20 at half-time, before Evrard Ourai's second reduced the deficit to just two points.
But May's effort and a late try from Tom Savage earned their side the win.
Bayonne: Jane; Laveau, Whitelock, Lacroix, Fuster; Meret, Duhalde; Iguiniz, Labouyrie, Choirat, Ducat, Jaulhac, Latimer, Lespinasse, Oulai.
Replacements: Leiataua, Bordenave, Taufa, Huete, Saubusse, Duputs, Maillard, Mamao.
Gloucester: Hook; Halaifonua, Scott, Atkinson, May; Twelvetrees, Heinz (c); Thomas, Hibbard, Doran-Jones; Savage, Latta, Ludlow, Rowan, Morgan.
Replacements: Matu'u, Orr, Afoa, Galarza, Kvesic, Braley, Thorley, Sharples.
Referee: Gary Conway (Ireland)
Assistant referees: Stuart Gaffikin and Richard Kerr (Both Ireland)
For the latest rugby union news follow @bbcrugbyunion on Twitter.
England winger Jonny May scored a try on his return after 10 months out with a knee injury to help Gloucester beat Bayonne in their Challenge Cup opener.