What kind of material should I use for rudder skins?
Some aircrafts have had fabric tails over the years that held up just fine. The Mooney Mite (M18) comes to mind. It topped out around 140Kts but flew just fine with a fabric covered empennage. The Vought F4U Corsair also had a full fabric empennage and operated well in excess of your quoted speeds so there is no problem using fabric for speeds you speak of. The FAA has a decent resource on fabric covering here thats worth reading through. 

The problem is less about interfering with yaw and perhaps interfering with everything else. Fabric is not as widely used anymore now that we have engine power to compensate for heavier airframes as well as the ability to work with metals consistently and accurately. Fabric also requires routine maintenance as well as periodic replacement. Fabric has a habit of ripping and needing repairs as well as simply sagging over time causing it to require a potentially full replacement. Since a great deal of aircraft are metal skinned now the people left to work on fabric coverings are slowly dwindling as well.