is it ok to use system_check_directory in a custom module?
Of course it is ok to use this function in a custom module as long as you take care of securing the form where this is used in. You should keep in mind that the error messages generated by this function could give valuable information about your file system to malicious users. If you limit access to the form that uses this function to trusted users only then there shouldn't be problem.

The function is tailored for core use, you can already see that in the way that it generates the .htaccess file depending on the #name attribute of the form array. But if you want to provide a configurable private directory to store uploaded files for a custom module then it seems like a convenient solution.