Why some cloths become rough after drying?
I believe it is down to water relaxing the fibres in the material. Upon drying in one position the fibres become temporarily fixed in that position. I have also read that the detergents used in washing can cause clothes to dry stiffer. A little experiment such as soaking your clothes in hot water should kill bacteria and then see if they dry stiff. This would give an idea as to whether or not its down to the detergent or just the water. 

Interestingly, a recent study suggests that fabric softeners reduce the wettability of materials. So if you use fabric softener on your towels, over time they will become less effective at drying (being towels).