Does iTunes have a limit on the number of movies it can list?
Be careful about creating a new library - make sure you have a backup first of the Library files first!

I've "lost" things in iTunes too.  Try creating a new playlist, and drag one of the missing movie files from the Finder right into the new playlist.  Even if it's already in iTunes, this will add it to the playlist.  If it's not in iTunes, this will add it to the Library and the playlist.

If it shows up in the playlist, but it's still not in the Movies section, check the "Media Kind" value on the Get Info -> Options tab.  Make sure it's set to "Movie", not "Home Video" or something else.

If it can't be added to the playlist at all (and really isn't somewhere else in iTunes already), it's probably in a format that iTunes doesn't support.  I know you said most of the HB-encoded movies are there, but that doesn't mean that some of them aren't somehow encoded (or maybe tagged) differently, and iTunes is choking on that.  Check the encoding settings carefully in QuickTime and VLC, and the tags in MetaX or the equivalent.