Can't understand how to invoke my own theme() function
You just need to set it on the $form element inside your form function:

function formexample_nameform($form, &amp;$form_state) {
  $form['#theme'] = 'formexmaple_nameform';
  // ...


When you use render element with a theme the variable on which you set #theme is the one that's passed through to the theme/preprocess functions.

Don't forget to add drupal_render_children() at the bottom of your template file or the form won't render correctly:

echo drupal_render_children($form);