Can I manually change an .mp4 to .m4v by changing the extension in the Finder (mac) or explorer (win)?
Yes.

M4V is a slight extension to the MP4 container format. M4V is Apple's version, which adds optional DRM-encryption.

So if the video is DRM-protected (i.e. came from the iTunes store) it should stay as m4v (and will open by default in iTunes). If it's not, it can be used with either extension, and many players will be happy either way. By default, .m4v files will open with iTunes, which may be convenient.