Swapping sugar for something less calorie dense but keeping the volume
Yes, the lack of volume will be a problem. Sugar makes up much of the structure of a cake. But since Stevia is 300 times sweetener than sugar, you'll end up using 300 times less. In addition the bulking issue, sugar plays many other roles in cake: it tenderizes, retains moisture, and contributes to browning. A sugar-free cake may be disappointing no matter what.
I also worry that you are making too many big substitutions. The recipe you are referencing was carefully developed to use wheat flour and real sugar: substituting vastly different products could lead to inferior results. Why not follow a recipe developed with your ingredients in mind? This recipe uses Stevia and almond flour as desired, but supplements with other binders such as flax and tapioca to ensure sufficient structure. Another uses Stevia, a gluten-free flour mix, buckwheat flour, eggs, applesauce, and flax. As you can see, to totally remove sugar and gluten from a recipe, you may need to carefully add other ingredients.
If you are really set on using an existing recipe, some recommendations for substitutions exist. This blog post prescribes replacing each cup of sugar with 1 teaspoon of Stevia plus some extra liquid. They give options:

applesauce, 100 percent fruit juice (apple, pineapple or grape are good choices), fruit puree, egg whites or plain, unsweetened yogurt. Fruit generally provides a lighter, fluffier texture, while yogurt gives the cake a rich, moist feel. Egg whites tend to lend density to cakes. Experiment with different substitutes until you find the one that best suits your tastes.

I haven't tried any of these recommendations, but let us know how your banana bread goes!