Adjusting baking powder to work with almond milk
Baking soda reacts with acid to create CO2 and lift.
Baking powder already has the necessary acid included.

Cow's milk is slightly acidic. Almond milk is slightly alkaline.
It may be that there is just a little bit less of the necessary acid to make your leavening react.

Experiment adding a little acid. Cream of tartar would be nice because it wouldn't throw off your liquid ratios or change the flavor. Lemon juice could also be used. I would start with 1/4 tsp of cream of tartar.

However, browsing around there are plenty of people online who don't seem to be having any trouble using almond milk in baking. It may be that the almond milk is a red herring and there are other variables that aren't controlled.
For example: Maybe the leavening was stale? Maybe the temperature of your oven was off?