Risk in evicting node from SQL Server cluster without prior uninstalling of SQL Server binaries
Decided to take a risk and test it in production.
Took out 2 nodes from 4 node cluster without uninstalling SQL from those and I can confirm for the two weeks of being in this situation I haven't receive any complaint from users.

Hope this helps others potentially dealing with same question