Can light affect specific objects only?
To remove specular, as that's the name for the thing pointed to, you select the faces that shouldn't have it and give them material of the same color, but with a different name. Then, in a node, set its Roughness to 1. This goes for Cycles.