How to structure T-SQL query with many left joined "attributes"
In the original Plan you have few Merge Join and then Nested Loops.

When the things slow down, you can see that plan is changed to: Hash Match (and parallel).

You can try to enforce using old plan by replacing inner join with inner MERGE join and inner LOOP join.

The documentation may help you dive into differences between them: Hints (Transact-SQL) - Join

Take note of the warning:


  If a join hint is specified for any two tables, the query optimizer automatically enforces the join order for all joined tables in the query, based on the position of the ON keywords.


If you do not wish to force the join order for all joined tables in the query, consider using OPTION (LOOP JOIN, MERGE JOIN) instead. This will allow the optimizer freedom to reorder joins and choose between nested loops and merge join on the basis of estimated cost.