How do I add a JavaScript file to a form?
There is no use to attach libraries to a form that is displayed in a modal window because the window has no place for assets (JS and CSS files). Instead I would suggest using hook_page_attachements or attaching the library to page element that is responsible for triggering that modal window.