Bete Noir -- no dairy
The key to a smooth ganache is fat - add too much water and you will end up with a “grainy” product. There are even recipes that use butter instead of cream (full or partial substitution), and while that’s probably a heart attack on a spoon, the texture is excellent. Almond milk is at least as “watery” as regular cows milk, so yes, that’s a questionable substitute without further changes.

You can benefit from the recent vegan trend and either use full-fat coconut milk (if the flavor is to your liking) or buy a vegan cream substitute, just check what your local store has available. If you want to use almond milk, consider adding a butter substitute (solid vegetable fat?). 

Margarine instead of butter in the crust won’t be an issue, alternatively, you may want to check vegetable shortening. We may have a few Q/As on that substitution on the site. And don’t forget to get dairy-free chocolate.