PWM Control, How to stop LED from Flickering?
Your code doesn't make much sense. 

You set the LED to 2 different brightness values that don't have anything to do with each other: The pot brightness and the button-based brightness. You only adjust the LED based on switches once every 5 mS. Thus the LED is going to flicker between 2 different brightness values.

What do you want it to do? How do you want to control the brightness based on input from both the pot and your switches? (What is the relationship between the buttons and the pot? Should the button input alter max/min value of the pot up and down? Should holding the button down change the brightness by 1 value per a specific time unit? Say 2 increments/decrements per second?)

Describe the user interface you want to achieve, and then think through how you would do that in code.