Best way to cover stair using concrete retaining wall block
I think they're nice looking stairs... I would suppose that it's not going to sit perfectly still without a stout concrete base, or without some kind of retaining system (like rail road ties or long blocks or something) but I think crushed rock is probably a suitable alternative, too. 



After a couple years, you will probably need to add something to fill it in. Then I might opt for a more ornamental gravel. 

Alternatively, you could go for the natural look if you fill it in with soil and grass seed; as grass and stuff begins to grow it will also help it settle and hold it's place. 



Since you have angled blocks/pavers... yo could try the (fancy) round stair look.