How to clean smell from granite mortar and pestle
Should be fine, in fact some people argue that a pestle and mortar is suppose to have remnants of past spices and that it adds to variety and such. 

If you're really concerned of the taste, smell or if you ground nuts (allergies), etc. or just don't want the smell/taste then soak it for a long period of time, then rewash it and repeat till it fades.