What is a better-tasting substitute for baking powder?
You might look at what the specific chemical agents are in your baking powder; it's possible that you have one that uses sodium acid pyrophosphate, which can leave a bitter taste.

Some people avoid the various aluminum based ones, as they can give a metallic taste, but it's likely preferable to bitter.

You can use baking soda and an acid, but it'll then be single-acting (it won't give a second rise when heated), which might affect your baked goods, particularly if you don't get it straight into the oven after mixing.