Rounded bevel for selected vertices in a plane
Option 1: Setting with Scroll wheel

If you use the scroll wheel while in bevel mode the bevel uses more edges, giving the corner a more rounded look.



Option 2: Using the advanced tool menu popup:

There is also another way (which doesn't need a scroll wheel) to set more advanced settings for the bevel function (actually works for any tool and add primitive actions).

After you did the tool action (in your case bevel) a small popup is seen in the left corner of the 3d-view screen (in Blender 2.80) which you can click to expand it. There you can modify the action you just did. 





PS: In Blender 2.79b this advanced tool menu was located in the tools tab



Option 3: Use the +/- keys

As quoted from the comment from @lemon you can also


  "use + or - from the keypad." - lemon


to increase the segments of the bevel tool.



Happy Blending! :)