Loop for finding proper grouping brackets
The most common approach to bracket matching is using a stack. 

Unmatched brackets you encounter are pushed onto the stack. When encountering a matching bracket pop the stack (and verify if it's the correct type of bracket, maybe not needed in your case). If the stack is empty after finding a match, the last match was the position you were looking for.

This approach, as you can see, also immediately allows for bracket matching validation, and can handle different types of brackets in the same String

To boot, the code is simpler.

Note

When looking for a stack implementation in Java, you may come across java.util.Stack; don't use that, use java.util.ArrayDeque instead, as the Stack javadoc itself suggests.