How to remove a listener IP from a SQL Availability Group Listener by T-SQL
Based on my experience with SQL Server 2014 and 2016, you can simply remove the IP address resource in Failover Cluster Administrator.  We have done this on several non-production clusters and on a production cluster in order to move from one subnet to the other.  I stopped the SQL Server service on all nodes to ensure that there would be no connections and less chance of some sort of conflict since it's a bit out of the ordinary.

The other option is to simply take a few minutes of downtime to remove the listener and rebuild it.  If you scripted all of the operations it should only take seconds.