I'm using steel cut oats flour to make bread and muffins but the insides are mushy and wet
This recipe is way too low in gluten, or even gluten free. You need wheat flour (or flour of other very closely related species such as spelt) to get a batter which can rise and bake normally. Any flour which does not contain gluten is not capable of trapping the bubbles created by the baking powder, and the moisture of the batter cannot escape the dough through the solid, bubbleless batter. The mashed banana and jam made it worse. 

The best thing to do would be to add pure gluten in proportions common to AP flour (9 g gluten to 91 g oat flour). You will get normal batter. 

If you don't want to do that, you can try to engineer a gluten free recipe using xanthan gum or other thickeners capable of producing a gloopy texture. The engineering process will take anywhere up to a dozen tests to get the texture right, and you will need to work with precision (normal scale to weigh all ingredients, small-amounts scale for the xanthan intself). Most home cooks prefer to use an existing recipe instead of going through this.