safety of using chamotte stone from hardware supplies store for baking
General consensus is that you should not use something that is not food safe for cooking.
And in particular something made for construction/building, it probably contains lot of non-food safe chemicals, and they are probably not obligated to list the ingredients; so you do not know what is inside.