Risks of manual failover in Always On Availability Groups
Do yo have a test environment?  If you don't, get a $200 free Azure credit and set your environment up there (same service pack level) and try it, it is one of the best way to gain confidence and find edge cases.  Nothing can really beat testing a like to like environment except experience.  Also check out the official support docs from Microsoft.

Per the doc you'll want to check out several items, including:

To determine the failover readiness of an secondary replica, query the is_failover_ready column in the sys.dm_hadr_database_cluster_states dynamic management view, or look at the Failover Readiness column of the Always On Group Dashboard.

To answer the question though, AlwaysOn failovers are very fast as the service is running on both instances and the listener then points to the new host.  Ensure the hosts are connecting to the listener instead of the IP of the machine.  

Thus, if tested and running properly in your environment it is much faster than FCIs (Failover Cluster Instances) during busy hours.  Note that AlwaysOn only works per database and not per server so you will want to ensure jobs are setup right on the new primary replica, users, permissions/sids, and if you need to maintain databases in sync you will want to ensure that you do that somehow before failing one of them over and not the other.  Also, if you use MSDTC for cross database transactions you'll want to be very careful as it can cause irreparable data corruption in many cases where you use cross db transactions within the same instance.

Perhaps Sean will be able to give you more of the issues you might face if he sees your thread.