How can I install a beta version of a module?
This installs the latest beta-12 for me:

composer require drupal/webform:5.0.*@dev


You need to do a bit of translation from the Drupal version number. From Using Composer to manage Drupal site dependencies:


  Specifying a version you can specify a version from the command line with:

 $ composer require drupal/&lt;modulename&gt;:&lt;version&gt; 

  
  For example:

 $ composer require drupal/ctools:3.0.0-alpha26
 $ composer require drupal/token:1.x-dev 

  
  In these examples, the composer version 3.0.0-alpha26 maps to the drupal.org version 8.x-3.0-alpha26 and 1.x-dev maps to 8.x-1.x branch on drupal.org.
  
  If you specify a branch, such as 1.x you must add -dev to the end of the version.


Hence for Webform, 8.5 becomes 5.0.x, and that's what you need to use to identify the package correctly.