Is there a way to set block visibility programmatically without using the PHP Filter module? By enabling the core PHP Filter module, I can use PHP code to set the visibility of a block in the block configure page.

But I don't want to enable the PHP filter module for security measure. So is there a way to do it in custom module? I can't find a hook doing so. And a close hook hook_block_view_alter can only change the block content.
Yes. Setting the blocks content to false will hide it from the display and will hide the region if there are no other blocks present. I'd imagine you would need a setup similar to the following:

$block['content'] = _my_module_custom_block();

...

function _my_module_custom_block(){
    if( HIDE_CONDITION )
        return false;
}