Do these materials typically contain asbestos?
I've bricked a few steel heat treating furnaces with asbestos bricks but never heard of asbestos mortar so I'd guess your safe there. It's crumbling a bit because it wasn't finished off correctly because it was in the attic. As far as the plaster, doesn't hurt to have it checked.