How do I use a custom theme in a module? As I know Drupal 8 removed theme(), which is still present in Drupal 7. There  is function hook_theme() to call and render the theme in a module.

Can anyone help with some of the examples? How do I use a custom theme in my custom module?
It is very easy. You just need use '#theme' element in your render array.

For example:

Drupal 7 has both way:

// Render array.
$build['cars_list'] = array(
  '#title' =&gt; t('Cars List'),
  '#theme' =&gt; 'item_list',
  '#items' =&gt; array('Mercedes', 'BMB'),
);
// Ready HTML.
$build['cars_list'] = theme('item_list', array('items' =&gt; array('Mercedes', 'BMB')));


Drupal 8 has only one way:

// Use render array.
$build['cars_list'] = [
  '#title' =&gt; t('Cars List'),
  '#theme' =&gt; 'item_list',
  '#items' =&gt; ['Mercedes', 'BMB'],
];


Actuality you may create a custom theme via hook_theme() and use it.