How to find Setup time and hold time for D flip flop?
Looking for the values of setup time that cause the FF to "fail to operate" is not a good idea. A FF will malfunction long before it starts to completely fail. This is a very optimistic specification for setup time, and it can lead to erroneous estimates of MTBF for metastability.

The effect of setup time on clock-to-Q delay is well known and has been documented for several kinds of flip-flops. Here's an example from 1996:


Foley, C., "Characterizing metastability," Symposium on Advanced Research in Asynchronous Circuits and Systems, 1996., pp.175,184, 18-21 Mar 1996, doi: 10.1109/ASYNC.1996.494449

Here's another example from 2001, regarding a \$0.25\mu m\$ process. In this figure you can see that the measured setup time is 190ps, allowing a 5% increase in clock-to-Q delay. If instead the setup time was estimated to be the smallest value that allows the flip-flop to operate the authors would have selected a much smaller value, about 120ps. However, this would lead to invalid timing analysis because when the setup time is reduced to 120ps the actual clock-to-Q delay has increased dramatically, from an nominal value of 1060ps to 1200ps. Of course, you could choose to specify the setup time to be 1200ps for timing analysis but then you have a longer critical path and a slower clock frequency.



Dejan Markovic, Borivoje Nikolic, and Robert Brodersen. 2001. Analysis and
design of low-energy flip-flops. In Proceedings of the 2001 International
Symposium on Low Power Electronics and Design, pp. 52-55. doi: 10.1145/383082.383093

Similar data for a 45nm process is in "Setup Time, Hold Time and Clock-to-Q Delay
Computation under Dynamic Supply Noise" by Okumura and Hashimoto, 2010. In "Multi-Corner, Energy-Delay Optimized, NBTI-Aware Flip-Flop Design" (Abrishami, Hatami, and Pedram in 2010 ISQED) the authors present data for a 65nm process with regard to both setup and hold time. The effect of setup time on clock-to-Q delay is often discussed in the important context of metastability, such as in "Comparative Analysis and Study of Metastability on High-Performance Flip-Flops" (Li, Chuang, and Sachdev in 2010 ISQED) which again uses a 65nm process. Another example is "Performance, Metastability, and Soft-Error Robustness Trade-offs for Flip-Flops in 40nm CMOS" by Rennie et al. in IEEE Trans. on Circuits and Systems, August 2012.

This is what I do. First, run a simulation with the D input changing one-half clock cycle before the relevant clock edge. Measure the clock-to-q delay under these conditions and consider that to be the nominal clock-to-q delay. Now start moving the D input transition closer and closer to the relevant clock edge, noticing that the clock-to-q delay will start to increase. When you have the D input edge at a point where the clock-to-q delay is 5% greater than nominal (or choose the percentage you like) then the time from D to clock is the FF setup time specification.

I didn't invent this technique. It is well documented in, for example, the HSPICE Applications Manual. In this manual, they refer to this effect as "pushout" of the clock-to-Q delay and state:


  For setup- or hold-time optimization analysis, a normal bisection
  method varies the input timing to find the point just before failure.
  At this point, delaying the input more results in failure, and the
  output does not transition. In pushout analysis, instead of finding
  the last point just before failure, the first successful output
  transition is used as the golden target. You can then apply a maximum
  allowed pushout time to decide if the subsequent results are
  classified as passes or failures. Finding the optimized pushout result
  is similar to a normal bisection because both are using a binary
  search to approach the desired solution. The main difference is the
  goal or the optimization criteria.


The manual also supplies this figure to explain the method:



The clock-to-q delay is very sensitive to the D-to-clock time as you get close to this point so I strongly recommend using a binary search (bisection) in your simulation to find the setup time. I know this seems like a lot of work but if you do it more than once you will be glad you invested the time. Learn how to automate measurements.

For a more in-depth discussion of why this is an important criteria for defining the setup time and some SPICE simulation examples, you might also want to look at Jos Budi Sulistyo's Master's Thesis from Virginia Tech ("On the Characterization of Library Cells", 2000). Sulistyo notes that (emphasis mine):


  Setup time is defined as the amount of time before the latching clock
  edge in which an input signal has to already reaches its expected
  value, so that the output signal will reach the expected logical value
  within a specific delay.


Deciding what is an acceptable increase in clock-to-Q (5%? 1%?) is a tricky business. Pick a small percentage and you get very conservative setup time values, which limit the maximum clock rate of your design. Pick a large percentage and you run the risk of introducing timing failures that the synthesizer can't see because it uses the nominal clock-to-q.

I interpreted your question to be specifically about running SPICE simulations to determine setup time. If instead you want to ask about the process of characterizing a standard cell flip-flop then we can talk about simulating over process corners, supply voltage, temperature, input slope, and output loading. These decisions are strongly influenced by what your synthesis tools need. For example, in an earlier life I generated characterization libraries for Synopsys' Design Compiler. There were separate libraries for different operating conditions and you included the appropriate set of libraries when performing static timing analysis. Each characterization library typically provided a 2-dimensional table for every timing parameter, showing variation due to input slope and output load. For the best results, setup and hold time should be characterized separately for a rising D input and a falling D input.