Media playback is not supported on this device
Tries by Tom Habberfield and Tyler Ardron, converted by Dan Biggar, gave the home team a 14-point interval lead.
Ulster dominated after the break, but Brendan Leonard's try saw the game off before Jacob Stockdale's late score.
Ospreys need a point in their final game to secure a play-off spot while Ulster will have to beat Leinster for any chance of making the final stages.
They face Scarlets on 6 May with the west Wales region secure in the play-off places after a 30-8 win over Connacht.
Fifth-placed Ulster will need a miracle to overtake to overtake Ospreys in fourth place and will regret the number of missed opportunities after the break in Swansea if - as now seems likely - they fail to make the play-offs.
Stockdale beat most of the defence, only to see his final pass go to ground and later kicked the ball over the dead-ball line after outstripping the cover.
Charles Piutau dropped the ball close to the Ospreys' line as Ulster put the home team under pressure.
But the visitors were made to pay for their profligacy when substitute Leonard crossed for the Ospreys' third try two minutes before time in a rare second-half attack.
When Stockdale scored practically from the re-start, it left Ulster looking for a second try and conversion to claim a losing bonus point - only to lose possession at a breakdown for the umpteenth time.
Ospreys - coming into the game after three successive league defeats - weathered an early storm and scored from practically their first attack as centre Ashley Beck tied in two defenders to send the unmarked Habberfield over in the corner.
Paddy Jackson and Biggar exchanged penalties before the Welsh side took control in the lead up to half-time with only a combination of good Ulster defence and their own lack of composure keeping Ospreys at bay.
But on the stroke of half-time a long kick by Jackson saw home full-back and man-of-the-match Dan Evans carve the defence open before handing on to Habberfield who put Ardron over between the posts.
The conversion opened a 14-point gap which better reflected Ospreys' dominance.
It was Ulster's turn to dominate in the second half, but mistakes and stubborn defending left them wondering what might have been - and facing an almighty challenge in the final game at home to Leinster.
Ulster scrum-half Ruan Pienaar, who leaves at the end of the season,  may have played his last game for the province after suffering a back spasm which saw him leave the field during the first half.
Ulster director of rugby Les Kiss said: "Frustrated and disappointed - two words that describe a lot of things because there's a fair bit of pain in that dressing room.
"Ruan [Pienaar] had a back spasm before the game started, he said he'd ride it out but within 20 minutes it was gone. It's a fairly significant spasm and those things can take a while.
"Stu McCloskey had a bad dead leg and Stuart Olding was hit hard with his first touch with an ankle injury, it's probably season-ending."
Ospreys: Dan Evans; Keelan Giles, Kieron Fonotia, Ashley Beck, Tom Habberfield, Dan Biggar, Rhys Webb (Capt); Nicky Smith, Scott Baldwin, Brian Mujati, Bradley Davies, Tyler Ardron, Sam Underhill, Justin Tipuric, James King.
Replacements: Scott Otten, Paul James, Rhodri Jones, Lloyd Ashley, Dan Baker, Brendon Leonard, Sam Davies, Josh Matavesi.
Ulster: C Gilroy, A Trimble (capt), L Marshall, S McCloskey, C Piutau, P Jackson, R Pienaar; A Warwick, R Best, W Herbst, K Treadwell, A O'Connor, I Henderson, S Reidy, R Diack.
Replacements: R Herring, C Black, R Ah You, F van der Merwe, N Timoney, P Marshall, S Olding, J Stockdale.
Referee: John Lacey (IRFU).
Assistant referees: George Clancy (IRFU), Ben Whitehouse (WRU).
TMO: Neil Paterson (SRU).
For the latest rugby union news follow @bbcrugbyunion on Twitter.
Ospreys survived a second-half onslaught by Ulster to all-but claim a place in the Pro12 play-offs.