How can I validate that I am passing an Asset into my custom twig filter?
Actually, you had your code just right imo:

use craft\elements\Asset;


...

if (!$asset instanceof Asset) {
    return false;
}