Wallet as daemon, what's the security-concern about "rpcallowip"?
if anyone guesses my server and credentials, he/she is welcome to mine
  for my wallet.


He/she can also call all other RPC commands. For example, sendtoaddress. So, if your wallet is unlocked you can loose your coins.