Slightly bitter potatoes or onion
My gut tells me that they are either under-done or need to be rinsed. Have you tried par boiling the  potato? I would try slicing the raw potato, boiling for a few minutes to draw out some of the starch - then rinsing them before adding to your recipe. Worth a try I think.

Good luck!