How to Use Ajax in form alter in Drupal 7?
I would be able to give you some detailed information about what you are doing wrong with hook_form_alter if you included some actual code from your custom module. However, since you are not giving any specifics, you can watch the video: How to build beautiful forms in Drupal 7 (note that I entered the URL so that it starts at the moment where the ajax is described). About 5 minutes of watching this will probably give you the answer to your question (since it describes exactly how to use ajax with hook_form_alter), but if you are really impatient, the actual code is shown somewhere around here. 

If you want to see a live, well-tested and documented example of a form being altered using ajax, there is one in the Examples module. You can find the actual hook here and the callback function here.

I hope this helps. If it doesn't, give us some more background about your code and we will be able to tell you exactly what you are doing wrong.