How do I choose the "date_popup" widget for custom date field from code?
If you install the version 2.x of the Date module, you will be able to use more form fields, including a "date_popup" field, which shows a pop-up to select the date.
The module has an option that allows the user to decide if the pop-up should be shown; if you want to override that option, you should use variable_set('date_popup_timepicker', 'default'). That is the default option used by the module, but if the user selects "none," then the pop-up will not appear.

About setting a custom format with the "date_popup" field, the form field uses the "#date_format" attribute that, in the development snapshot 7.x-2.x, is said to accept most of the formats. The default value for that attribute is the one returned by variable_get('date_format_short', 'm/d/Y - H:i').
The complete list of the attributes used by the form field, reported in the comment for date_popup_element_info(), is the following one:


"#date_timezone"
The local timezone to be used to create this date.
"#date_format"
Unlike earlier versions of this popup, most formats will work.
"#date_increment"
Increment minutes and seconds by this amount, default is 1.
"#date_year_range"
The number of years to go back and forward in a year selector, default is -3:+3 (3 back and 3 forward).


Drupal core modules have only the "date" form field, and it doesn't use any "#widget" attribute. Third-party modules could define new attributes, though; as far as I know, no module defines "#widget" as attribute for form fields.