Aluminum and aluminum oxide
I am not an expert in food safety, so understand this is for information only.

Aluminum, when exposed to the atmosphere, will immediately oxidize.  Aluminum oxide is not very soluble an will likely pass straight out of the body when ingested.  Aluminum oxide is in certain toothpastes as an abrasive.  

If aluminum is exposed to fluoride, chloride, or sulfate there could be an issue if it converts to a soluble form.  As long as grinders are cleaned according to the manufacturer's recommendations this should not be an issue.     

The amount of aluminum oxide which wears off a grinder is related to the abrasiveness of the material being ground.  Stainless steel wears about six times less than aluminum.  A titanium grinder is even better.