Problem with displacement maps
The source for the displacement map in this case has to be an image or procedural texture, not the model itself. One way to do what you want would be to get or create tileable plaster texture first (look for "tile plaster texture bump" on a search engine), then create a material just for the walls, assign it and unwrapp the walls as you see fit.