SQLPlus configuration problem or firewall issue
I end up finding the answer :)
On my test server on the same network as the DB I made a tcpdump and I notice that my connection started on the primary IP and the DB reply through the secondary IP. There was a request on the DNS server in the middle.
I made the same test on my ansible-test server and found out that there was problem on then DNS request and fail the connection. Basically, it was not able to understand the secondary IP name.
So this end up being a weird networking problem.
Thanks All.