howto: limit movies displayed in itunes "in-app" movies folder to those in the itunes library
iTunes is going to display any files that are in its library. To avoid displaying files, don't import them into iTunes and don't use iTunes as the player for those files because they will be added to the library collection. To play those files use Quicktime Player or a third-party solution like VLC instead.

You can also disable iTunes's copying of files into it's own folder structure in iTunes > Preferences > Advanced, but this will apply to all media types and, in my experience, is a less-than-optimal way to use iTunes.