Rabbit hole: what's the recommended setting for content that should not be directly accessible? I have one content type, as well as several vocabularies (taxonomy) that are only shown on the site through views and therefore should not be accessible on their own. I'm using the rabbit hole module to prevent that from happening.

My question is this: from an SEO point of view, is it better to use a 301 redirect (to the front page), or a "page not found" response?

Maybe I should point out that the site isn't online yet, so it hasn't been crawled by search engines.
It's really up to you what you want to do but the way I see it, if a user goes to a random non-existing path on your site they will get a 404 error, so if the rabbit hole page is a page the user should never be able to get to then it should also be a 404 error because as far as that user is concerned it is not a valid page.

Now if you redirect all 404 errors to your home page or search page or something then these pages should have the same behavior.

If the page is something that anonymous users cannot see but a paying member can (or some similar scenario), then it may make more sense to set rabbit hole to a 403 error, because it is a page that they can't currently see but could see if they log in.