What is the best method to determine bypass capacitors for each chip in Mixed-Signal Systems?
It depends. There are no universal calculations to answer your question. Well, except for SMPS circuits where the core functionality is voltage ripple and ripple currents are significant. 

One way of viewing the power plane is as a power net impedance, trying to minimize the impedance allowing best possible reaction time from the components with minimum ground bounce. The other approach is to see power plane as a bucket of electrons that should not wobble too much with switching circuitry. 

These approaches are not mutually exclusive as large caps reduce voltage ripple and small caps are good for high frequency impedance. 

One very important thing to remember when you connect different size capacitors in paraller is that this creates unexpected impedance spikes. So in general you have to be careful not to create a high-impedance zone on a circuit load frequency. 

Most people just plop 100nF bypass cap on each power pin and add a larger (10uF or similar) cap farther away from a major component such as FPGA or a high-speed ADC. For high-speed circuits you may want to use a tool such as murata simsurfing to pick a specific capacitor physical size/value to create an resonant dip on the fundamental frequency. Note that tuning the capacitor in this way allows the circuit to switch faster which may in turn create worse EMI. 

And as a final note, always connect bypass caps right up to the power pin, not five millimeters away, if at all possible. It's ideal to connect to both GND and VCC pins this way but you may stick the other end of the cap and GND pin to GND plane with a via as necessary. Having the bypass caps on the other side of the PCB degrades performance as there's the additional via inductance but this becomes signifant only with increasing frequencies.