What causes SQL error 41106?
The default database for my login was set to 'TempDB'. Changed that to 'Master'. And there was a DNS registration issue with the first node. So the secondary replica wasn't able to communicate with the primary by computername. Correcting these two resolved the issue. 

Join-SqlAvailabilityGroup worked properly.