How to remove modules via ftp?
You need to disable the module on http://example.com/admin/modules first. That should take care of your error messages. 

If your administration interface is not accessible - caused by the error message - you could access the database directly and disable the module in the system table - as suggested by Mołot and outlined in https://drupal.org/node/157632. 

Additionally to disabling the module, you should also uninstall it - which will remove any configuration and data associated with this module from the database: http://example.com/admin/modules/uninstall (at least it should, if it is properly written). 

Now, you can safely remove the module from the modules directory, if you want to. Since the module is disabled and uninstalled, it will not interfere with your Drupal installation, however, it is advisable to clean up the modules directory, and move the module to a location outside the webroot - to avoid issues if the module has security holes and the web-server is (mis)configured to allow direct execution of modules. 

Before you do all this, I would save a backup of the database and the custom module, in case the functionality and data might be needed after all.

You can of course do all of this also on the command line using drush - since you asked about ftp, I assume you do not have ssh access.