How can instructors add users to their classes and at the same time not know all of the institution users?
Yes you could add three additional tables called Departments, DepartmentUsers and DepartmentClasses.
Departments would have a primary key field departmentId (and probably some other fields like departmentName, etc).
DepartmentUsers would be a linking table between Departments and InstitutionUsers so it would store the departmentId and institutionUserId (as foreign keys). (This table determines which users are in which department.)
DepartmentClasses would be the same as above, except it would link Departments to InstitutionClasses, so it would store the departmentId and institutionClassId. (This table determines which classes are in which department.)
Then for a particular class you can see only the applicable users (within the department) who can be added to it via InstitutionClasses join to DepartmentClasses join to Department join to DepartmentUsers.
(Then the institutionUserID field in your InstitutionClasses table only represents who's already been added to the class. You may want to rename the column to something more explicit like registeredInstitutionUserID.)