Do I need to worry about voltage, coming from ground, and effecting an analog input?
The arduino has build in pull up resistors, it is not a residue voltage coming from ground more than a "not 100% 0 volt" resistor.

Even though there is built in pull-up resistors one can add additional resistance to the arduino to make sure it will not show a false positive.

From what i have learned, you cannot experience voltage comming from ground (if everything is properly wired), if you want to make sure. Place a diod in the direction of the power flow, to prevent this.