Avoiding "Zoom bombing" in virtual conferences and seminars
How about using something other than Zoom? Other softwares support features that can help with this. Moreover there are some serious security and privacy concerns about Zoom (see e.g. this statement by the FBI and this investigation by the NY attorney general; Bruce Schneier has written an overview of the concerns here).

In our department we use BigBlueButton (website, Wikipedia) for our classrooms and seminars. It's a completely open-source and free web conferencing system. There are a few services out there that host servers but they're completely overloaded at the moment. I'd say your best bet would be to get your IT services to host a server for your department. With BBB by default new users don't have their webcams activated, and you can prevent them from doing so. You can even prevent them from using their microphone and require them to virtually "raise their hand" before a moderator allows them to speak. That should cut down a lot of the abuse. You can also set up the service so that an account is required before accessing the meeting. Unless the abuser is very dedicated, they probably won't bother creating dozens of accounts to continue bothering people after being kicked out, I hope.

Also, something obvious: many abusers are specifically targeting Zoom and going into random Zoom meetings. Unless you've somehow attracted abusers who are specifically targeting you, if you use something else (especially something hosted by your own department), you probably won't have any problems.

Note: I'm only recommending BBB because that's what we use at our department, but there are other services out there. I know another department has set up something called "Panopto" for example, and another something called "Classilio" - you'll have to look that up, I don't know much about it. For informal meetings we use Jitsi but it doesn't seem adequate for hosting a seminar. But in any case, if a software doesn't meet your requirements, you should definitely look into switching to something else, especially with the privacy concerns...