Red Sign of Shudde M'ell hands use and timing against Nightmarish X in Arkham Horror
Red Sign can be cast during any phase (so combat during any phase isn't an issue).
'Combat' begins as soon as you stop at/leave an location/street with a monster or during the encounter phase when 'A monster appears!', etc. This is to say that you're already in combat when you're figuring out how many dice to roll for your horror check. As Red Sign makes no specific mention of casting while making a combat check, rather it only refers to 'combat' in the general sense, Red Sign can be cast before making a horror check to remove Nightmarish from a monster.
You pose an interesting situation with your second question, but this has nothing to do with your primary question and is covered by the other question you posed here:
Stateful equipment use for Red Sign of Shudde M&#39;ell in Arkham Horror
Additionally, from the Kevin Wilson thread of the FFG forums:

Can Enchant Weapon, Shrivelling, Wither, Dread Curse of Azathoth, and Red Sign of Shudde M'ell be cast at any time after an investigator engages a monster, so as to, for example, use the Red Sign to cancel Nightmarish before the Horror Check, or allow one to see if a &quot;weapon&quot; type spell succeeds before making the decision to fight or flee?
Yes. Yes you can.