how to compare two groups
I assume you are performing an independent samples t-test. That the Ns are different isn't necessarily a problem -- the mean is an unbiased estimator -- but just how different are the sample sizes?
As you describe, you are probably violating the homogeneity of variance assumption. 

T-tests tend to be fairly robust to this violation, but you might consider having some outlier treatment and/or using a nonparametric test.