Stay connected to specific Wifi which has no internet
I found a possible solution for my problem.
I simply set the DHCP settings to static, give the devices an appropriate IP and Gateway, and left the DNS preset as it is (8.8.8.8) by following these instructions: Is it possible to assign a static IP address without a DNS?
Afterwards it still complains about the missing internet connection, but this time it keeps the connection.
I tested it on both my Nexus 5 and Nexus 9, and it seems to work.