Winglet design and airfoil selection
The comments explain the logic behind a winglet well, so I will focus on the remaining questions:


  what would be the optimal airfoil selection for the winglet?


As a first rule of thumb, continue with the wing airfoil. A cambered winglet will be most efficient, because winglets are only beneficial if they produce lift. How much? This depends on the ratio of wing tip chord to winglet root chord. Ideally, there should be no break in the spanwise distribution of circulation, so the product of local wing chord and local lift coefficient should be the same at the transition and taper out towards the winglet tip.


  would the geometric twist affect the performance of the winglet?


Yes: The incidence must be chosen such that the desired lift coefficient is trimmed at the operating point (which is the cruise condition in case of airliners). Twist will only be needed when the planform differs a lot from the elliptical distribution. When flying with a high wing loading (again, just as airliners do), be sure to include the effects of aeroelastic deformation: They can be quite severe around the wing tips in cruise.