How to sanitize filename in D6?
Well it depends on what you want to allow in your filenames.  There is a similar question here where you will find how to sanitize filenames in PHP.

From the above link, I had used following function to sanitize filenames in my one of the modules:

function mymodule_sanitize_string($string) {
    $string = htmlentities($string, ENT_QUOTES, 'UTF-8');
    $string = preg_replace('~&amp;([a-z]{1,2})(acute|cedil|circ|grave|lig|orn|ring|slash|th|tilde|uml);~i', '$1', $string);
    $string = html_entity_decode($string, ENT_QUOTES, 'UTF-8');
    $string = preg_replace(array('~[^0-9a-z_-]~i', '~[ ]+~'), ' ', $string);

    return trim($string, ' -');
}


If you want to sanitize user input then Drupal has many input filtering functions  e.g. check_plain(), filter_xss(), check_url() etc. There is a good tutorial on Drupal security essentials here.