How can I tell when InnoDB recovery is complete/check its progress? I'm writing a process to bring up a new instance from our nightly database snapshots, and I want to watch its progress and perform some actions once the database is up. When mysql first comes up, it spends some time in recovery, and I'd like to know when this is complete, and preferably how far along it is.

Linux "service" claims mysql is in "start/running" state, so that's no help. I can just try to connect until a connection is successful, but then I could be hung forever if something goes wrong during recovery. The closest I've been able to get so far is to tail MySQL's error.log, then use a regex to pull the last number off the last line to determine recovery progress. But that seems a bit fragile and hacky. 

Is there any sort of tool MySQL (or third party) provides to know for sure that MySQL is currently in recovery and how far into the process it is?
The right way to get MySQL status regarding recovery is monitoring its error log. There are 3 phases on InnoDB recovery:


Checking partially written pages (very fast)
REDOing committed and uncommitted transactions written to the transaction log. This can take some time, but its time is IO bound (as fast as reading/writing to the transaction log). If you are concerned about recovery time -and not about writing performance-, you can reduce the transaction log file size (or put those files on a faster media). After this, InnoDB will allow MySQL start accepting connections, so the server will continue as usual.
Rollbacking uncommitted transactions (potentially very slow). For example, if an alter table failed, it will have to rewrite all modified pages to its original state, but the server is up and running. The larger is your buffer pool, the faster this will execute.


All this phases are marked on the log.

There are not many 3rd party tools to monitor this. One I can think about is Percona XtraBackup, that provides you a percentage of the replayed log  but a) this is a backup/recovery tool for InnoDB and b) it does its own recovery process, independent of the mysql service (you cannot run both at the same time). It may not be what you are looking for, but it may allow you to do the recovery with a copy of the data to make sure it runs before starting mysql again.

Having said that, InnoDB has been very resilient to me, and the only reasons why recovery was to be done in a more "forced" level is because of harware/filesystem level corruption, or InnnoDB misconfiguration, never on normal operation. A different thing would be it taking a long time -rollbacking can be very expensive, an in some cases it is faster to recover from a backup than doing everything transactionally (DROP large_table).