How can I fill a small tapered gap between my pavers and a step?
You just lay the carpet or rubber sheet on the pavers (not attached) and use the vibrator. 

As for the sidewalk being lower than the dirt by about 4-5 inches, you could put in a retaining wall/edge, or make a box planter into an edge (a french drain may be helpful, here), or shape the soil so that it diverts/drains away into a swale (aka a ditch). 

To help keep pavers put (because you're worried about pea gravel in the sliver), use paving edger:



For the concrete idea, here's a couple of videos: Pebble inlay in concrete and Stone pebble mosaic ideas. Another method involves simply exposing aggregate by basically washing off the surface of the concrete.