How to extend flood control in a custom module?
You need to do 3 things when you validate your password entry form:


Use flood_is_allowed() to check that the user hasn't already breached their flood limit and set a form error with form_set_error() if they have.
Use flood_register_event() to log failed attempts on incorrect password errors.
Use flood_clear_event() when the user enters the correct password.


You can also decide whether to do it based on uid, ipaddress, both of those, or something else entirely.

You can see the example of the user module in user_login_authenticate_validate() and user_login_final_validate(). That example may or may not be more complex than you need in your specific case.