Does leaving the Views UI module enabled affect performance?
Every enabled module affect performance, just for the fact they are loaded, and parsed from PHP. views_ui.module, in the branch 6.x-3.x-dev of Views, is 10355 bytes.
There are then hooks that are invoked from Drupal, even if you are not using anything from the Views UI module, such as views_ui_help(), views_ui_menu(), views_ui_theme(), or views_ui_library_alter().

Another reason to disable unused modules is avoiding possible security issues.