How do I displace only outside of cylinder and exclude interior of cylinder mesh?
Create your cylinder with a reasonable amount of subdivision, (which will eventually separate vertex groups), and give it 2 vertex groups; mine are called 'Shell' and 'Rim', for convenience. You don't need to allocate to them: they're placeholders.



Give the cylinder a Solidify modifier. In the modifier there's an 'Output Vertex Groups' section. There, you can set 'Shell' to Shell, and 'Rim' to Rim.
The resulting 'Shell' vertex group is shown on the left, 'Rim' on the right.



In this case, using a Vertex Weight Mix modifier, I added the 'Shell' group to the 'Rim' group. This was to isolate a region (the blue one, below) from the rim, which would displace more neatly. I have a Bevel (by angle) modifier below the Solidify, to harden the edges of the rim of the roller.



Now, you can aim the Displacement at the inverse of the mixed group, (using the little double arrow) to displace only in that outside region.


With this sort of result:

This keeps all your modifiers live. But, as @moonboots has commented, it may be simpler, for this project, to work more destructively, applying modifiers as you go along.