Transforming the case of text using Excel VBA
First, I'd get rid of the modify button. You can use option button Click events to handle the change.  Something like this:

Private Sub opbLower_Click()
    ActiveCell = LCase(ActiveCell)
End Sub

Private Sub opbProper_Click()
    ActiveCell = Application.Proper(ActiveCell)
End Sub

Private Sub opbUpper_Click()
    ActiveCell = UCase(ActiveCell)
End Sub


To answer your question, you could also use the click events to return a format type for your btnmodify_click method to handle by creating a non local variable above all the code like this:

Dim format As String

Private Sub opbLower_Click()
    format = "lcase"
End Sub

Private Sub opbProper_Click()
    format = "ucase"
End Sub

Private Sub opbUpper_Click()
    format = "pcase"
End Sub


Then apply the condition in the btnmodify_click method based on the value of format.  The first example is better though, it involves less work.