Updating custom modules
"Updating a module" can comprise of two things:


If it's just code, then simply replacing the old code with the new code should be sufficient. However, when your code is complex or you are working in a multi-user environment, it is best to bring your code under version control like svn or git (as you have tagged), and maintain it with branches and tags depending on the version of Drupal that you are working with.
If your module also involved database updates, then you will need to become familiar both with the .install file as well as hook_update_n() functions. The documentation for the latter along with source examples from the .install files of core modules should guide you to creating update functions correctly.


On a related note, if your custom code is generic enough to be of use to the rest of the community, you can post it as a Drupal project and leverage drupal.org's project management feature which comes with git access.

You should also test everything out on a development server rather than on the production site. Adding lots of documentation to your update functions as well as maintaining a CHANGELOG.txt is also recommended.