Why does SQL Server not perform constant (UNION ALL) branch elimination with OPTION(RECOMPILE) when selecting the result into a scalar variable?
Constant folding has a particular meaning in SQL Server. It is not directly involved in your question. The features that combine to produce extensive simplification of the execution plan for you are the parameter embedding optimization (PEO) and contradiction detection.
PEO embeds the literal value of e.g. a parameter or local variable into the query text, in circumstances where it is safe to do so. One of the requirements is that OPTION (RECOMPILE) is specified. This guarantees that the plan generated will never be reused, so it may be safe to replace non-literals with sniffed literals.
The OPTION (RECOMPILE) hint itself only provides that a new plan will be generated on each execution, the sniffed values of any parameters will be used for cardinality estimation, and the one-off plan generated will not be cached for reuse after execution.
PEO was first added to the product in SQL Server 2008, though it was disabled shortly afterwards due to the possibility of incorrect results. It was re-enabled in SQL Server 2008 SP1 CU5 (Microsoft blog post).
A query optimized with PEO applied appears to the query optimizer exactly as if the query had been written with literals instead of parameters or variables. Contradiction detection can remove entire clauses or relational operator subtrees where literal expressions like WHERE 0 = 1 appear. This facility exists because automated tools often generate such SQL.
It is not always safe to apply PEO, but the exceptions are not officially documented. One exception is where variable assignment occurs (others exist, such as where the parameter appears in an OPTIMIZE FOR clause). My understanding is that variable assignment involves a good deal of complex legacy behaviour, with occasionally odd semantics, preserved for backwards compatibility reasons. It would be impractical to guarantee that PEO would operate correctly in all circumstances, so it is disabled for that case.
PEO is an opportunistic facility that goes beyond the documented behaviour of OPTION (RECOMPILE). It can deliver significant performance benefits in many cases, but it is not officially documented. One might usefully regard it as a bonus feature - nice when you get it, but there are no refunds in case of disappointment.
In your example, where PEO cannot be applied, elimination of subtree execution is provided by start-up filters (which are documented). The Filter operators shown in the &quot;unoptimized&quot; plan are start-up filters that only execute their subtree if the start-up predicate evaluates to true.
The lack of a &quot;seek plan&quot;, in the PEO context, is often due to optimizations that can only be performed (either due to safety issues or implementation limitations) when a literal value is present. This literal may appear in the original text, or it may have been substituted in via PEO. An example of this is the optimization rule SelOnSeqPrj, which allows a predicate to move past a sequence function like ROW_NUMBER when safe, but only when a literal value is available.
A SQL Server 2017 repro of the code in the question does not produce the extra Compute Scalar and implicit conversion mentioned in the question. It appears the query used to produce that plan was different to that given in the question. Or perhaps the instance or database has some important configuration or option that was not specified. In any case, I was unable to reproduce it.
The OPTION (RECOMPILE) hint is never ignored. The only implicit conversion in the query is from the internal bigint result of COUNT(*) to integer as required by COUNT (as opposed to COUNT_BIG).
Depending on the requirements and restrictions of the real application behind the question, you may need to employ dynamic SQL, or some other solution. Feel free to ask a new question about potential solutions to the underlying problem, if it can be expressed in a way that is suitable for our Q &amp; A format.

The brief answers to your questions are:

Yes.
I cover it in Parameter Sniffing, Embedding, and the RECOMPILE Options.
No.