Should I drill through the concrete garage wall to install a generator inlet switch inside?
Do the right thing not the easy thing. The question here is, how much longer will the feed line be? if its just a few feet then there is no problem. What you want to avoid is a long extension between the generator and the house.  Long line = voltage drop. And there is cosmetics. Think how a professional would do the install and act acordingly.