The Pirates were bottom after losing their first two games but Tom Lockyer headed them into a 55th-minute lead before Billy Bodin struck twice to secure their first points of the season.
Jermaine Beckford and Nicky Ajose scored consolations in stoppage time but the scoreline flattered a woeful Bury side.
Beckford and Harry Bunn tested Sam Slocombe in the first half but otherwise Rovers played some fine football, with Bodin firing wide as they racked up 26 shots.
Ellis Harrison and Tom Nichols were then denied by Bury keeper Joe Murphy before Chris Lines fired over.
Nichols and Ollie Clarke also went close before the break but 22-year-old centre-back Lockyer - in his 200th appearance for Rovers - then headed home from Lee Brown's corner.
The hosts offered little in reply and Bodin headed home from Joe Partington's 72nd-minute corner before poking in from close range after 86 minutes.
Beckford headed home from Tsun Dai's late corner while Ajose converted Greg Leigh's cross in stoppage time but Bury could not avoid a second straight defeat.
Match report supplied by the Press Association.
Match ends, Bury 2, Bristol Rovers 3.
Second Half ends, Bury 2, Bristol Rovers 3.
Goal!  Bury 2, Bristol Rovers 3. Nicky Ajose (Bury) right footed shot from the centre of the box to the bottom right corner. Assisted by Greg Leigh.
Tsun Dai (Bury) wins a free kick in the defensive half.
Foul by Liam Sercombe (Bristol Rovers).
Goal!  Bury 1, Bristol Rovers 3. Jermaine Beckford (Bury) header from very close range to the bottom right corner. Assisted by Tsun Dai with a cross following a corner.
Corner,  Bury. Conceded by Ryan Sweeney.
Attempt saved. Liam Sercombe (Bristol Rovers) right footed shot from outside the box is saved in the top centre of the goal.
Goal!  Bury 0, Bristol Rovers 3. Billy Bodin (Bristol Rovers) right footed shot from the centre of the box to the bottom left corner. Assisted by Lee Brown following a set piece situation.
Substitution, Bristol Rovers. Dominic Telford replaces Tom Nichols because of an injury.
Corner,  Bury. Conceded by Liam Sercombe.
Attempt missed. Tom Nichols (Bristol Rovers) right footed shot from very close range is too high.
Attempt saved. Rory Gaffney (Bristol Rovers) left footed shot from the right side of the box is saved in the bottom left corner.
Attempt saved. Tom Nichols (Bristol Rovers) right footed shot from the centre of the box is saved in the bottom left corner.
Foul by Callum Reilly (Bury).
Dominic Telford (Bristol Rovers) wins a free kick in the attacking half.
Hand ball by Rory Gaffney (Bristol Rovers).
Goal!  Bury 0, Bristol Rovers 2. Billy Bodin (Bristol Rovers) header from very close range to the top left corner. Assisted by Joe Partington.
Substitution, Bristol Rovers. Stuart Sinclair replaces Chris Lines.
Substitution, Bristol Rovers. Rory Gaffney replaces Ellis Harrison.
Foul by Zeli Ismail (Bury).
Liam Sercombe (Bristol Rovers) wins a free kick in the attacking half.
Harry Bunn (Bury) wins a free kick in the attacking half.
Foul by Ollie Clarke (Bristol Rovers).
Attempt saved. Ellis Harrison (Bristol Rovers) right footed shot from outside the box is saved in the top centre of the goal.
Attempt blocked. Ellis Harrison (Bristol Rovers) left footed shot from outside the box is blocked.
Attempt missed. Liam Sercombe (Bristol Rovers) left footed shot from outside the box is close, but misses to the left.
Substitution, Bury. Zeli Ismail replaces Alex Bruce.
Goal!  Bury 0, Bristol Rovers 1. Tom Lockyer (Bristol Rovers) header from the centre of the box to the top right corner. Assisted by Lee Brown with a cross following a corner.
Corner,  Bristol Rovers. Conceded by Alex Whitmore.
Hand ball by Jermaine Beckford (Bury).
Foul by Callum Reilly (Bury).
Ollie Clarke (Bristol Rovers) wins a free kick in the defensive half.
Attempt blocked. Callum Reilly (Bury) right footed shot from outside the box is blocked.
Foul by Chris Humphrey (Bury).
Lee Brown (Bristol Rovers) wins a free kick on the left wing.
Attempt missed. Harry Bunn (Bury) right footed shot from outside the box is close, but misses to the left.
Second Half begins Bury 0, Bristol Rovers 0.
Substitution, Bury. Chris Humphrey replaces Craig Jones.
Substitution, Bury. Tsun Dai replaces Callum Styles.
Bristol Rovers brushed aside their poor start in League One by cruising to an impressive 3-2 win at Bury.