Does AES-GCM without security tag turn into regular CTR Mode?
I'll keep to the final question as security evaluating the entire scheme would be off topic:


  If we use the implemented AES-GCM mode but without security tag (TAG_SIZE 0 bit) does this make the cipher behave like regular CTR mode as to the attacker can no longer start guessing passwords and knowing whether he has guessed the correct one and thereby losing this potential oracle?


Yes, GCM would then just get you CTR mode, but with a specific way of calculating the initial counter: $n &lt;&lt; 32 + 2$ where $n$ is the encoded nonce, shifted left 32 bits (multiplied with $2^{32}$), and then added the value 2 in the zeroed, least significant bits. The proof is in the implementation.



Notes:


The implementation I pointed at uses a (GCM default) nonce of 12 bytes; if you do not do that then calculating the initial counter will get more complex.
Not all libraries will allow a tag size of zero as that obviously falls outside the acceptable range for the size of the authentication tag $t$.




Off topic, some remarks on using GCM in your protocol:

As others have commented, creating (or in this case hacking) your own protocol is probably not the safest thing to do; there are already schemes out there that will do this for you. And just using CTR instead of GCM should be preferred if you go with this scheme.

Beware that the decryption will always succeed, but that parsing / handling the plaintext may still result in an oracle. E.g. if you encode a number and send it then decoding could possibly fail, resulting in an oracle.