Edible straws that look like plastic ones
Pirouette cookies (here by Pepperidge Farm) are rolled wafer cookies. The inside is stuffed with chocolate, chocolate hazelnut or vanilla.

Here's a recipe that includes a picture using them in a manner similar to what you're asking.



You could follow the recipe, but tweak it to look more like a straw and with colors and flavors that suit your purpose.