Remove known Excel passwords with PowerShell
Once you've found the right password and moved the file, then there is no need to try the rest of the passwords on it. You can break out of the loop with a break command.

(You have to change the pipeline % into a foreach ( in ) because otherwise the break would break out of the outer pipeline too. Thanks to Lieven in the comments for that.)

foreach ($passwd in $arrPasswords)
{
    # ...

    # if password is correct save decrypted file to $decrypted_Path
    if ($Workbook.Worksheets.count -ne 0 ) 
    {   
        # ...
        break; 
    }

    # ...
}


That may or may not speed things up.

Otherwise, I haven't tried them but I understand that there are .NET Excel libraries that you can use to manipulate Excel files without having to use Excel itself. COM automation is pretty slow, so using one of these libraries might be faster.