With a HDL design how/when do we know that we need multi_cycle path, how do we implement it
You've asked a number of very general questions, so my answers will necessarily be very general as well. If you have a specific example you'd like to discuss, add it to your question.


  With a HDL design how can one predict that the combinational logic between two registers shall be more than clock cycle and thus there is need to use the multi_cycle path constraint?


It is implicit in the design. If you require the result to be available in one clock cycle (as specified by the behavior), the clock cycle will need to be as long as required to allow this to happen. If the behavior is written so as to allow 2 or 3 clock cycles for the result, then this is when you also need to add the multi-cycle constraint. You don't do the latter without also doing the former.


  Besides this, for a HDL design where we may not know exactly which registers we are talking about, how can we put in a multi_cycle path constraint?


At this point, you need to abandon the high-level abstraction and dig down into the implementation details as provided by the synthesis tools. It can be tricky at first to decipher the reports, but it becomes easier with experience.


  Is it important that this delay be more than 1 clock cycle under best as well as worst conditions?


No.


  As far as I understand. We first synthesize our design and if the propagation delay of the gates is big enough between two registers than we know we need this constraint. This shall happen before the fitter as after fitting the routing delay shall add up as well. Is this correct?


Sort of. The synthesis tools will work hard to implement the logic so that it meets the timing constraints you have already established, including routing delays. If it can't, it will tell you.

Advanced tools even know how to move logic from one side of a register to another in order to balance the propagation delays, minimizing the clock period required.


  Can we also force the fitter to fit logic in such a way that it takes 2 clock cycles for data to reach the latch register?


In general, no. But there's no real need to do this.


  My confusion exists because when we have a HDL design we are at a higher level of abstraction and don't know about all the physical registers that exist in the design, that is unless and until we do a STA on the post-synthesis netlist.


High levels of abstraction a fine to a point, but if you really care about getting the most performance out of your design, you need to be willing to work down at the RTL (register-transfer level) code, in which you do explicitly know where the registers are and what logic goes between them.

My own background is that of an EE, so I always write my synthesizable code as RTL, since that's how I think about the hardware design. I only use higher-level, purely behavioral code in my simulation testbenches, which are never intended to be synthesized.