Predicting Execution Time of DSP programs I've been tasked with predicting the performance of our DSP code, for the obvious reasons (reduce time to market, add certainty in choosing chips, etc.)

Consider a system composed of many subsystems with timing results $P_{i}$ (in seconds) (filters, gains, dynamics processing etc.) with a clockrate $C$ (seconds/sample) and sampling rate $Fs$ (samples/second)

Our initial assumption is that:

$$\text{CPU utilization} = 100\% * \sum_{i}^{i=N}{P_{i}*Fs/C} \stackrel{!}{=}
\texttt{top -d 1}$$ 

i.e. CPU utilization estimate should equal the output of top -d 1`.

However, this doesn't hold up to real data! It's always underestimated.

We measure individual subsystems by wrapping each of them in a test harness and timing their execution. Our system runs in a single thread and on a single core.

Depending on the platform, we can get within 5 to 20%, but always underestimating top.

Why is this?

I can find nothing in the performance literature that lines up with our assumption, but the explanations are so numerous that it's almost like "we don't really know why we can't predict performance."

I am beginning to think that the solution is monitoring like a hawk during early development and scaling back the system complexity as we uncover new solutions.

However, it would be pretty awesome if we could design a system with the performance requirements ahead of time. Our requirements have largely been determined by "this is better than the old thing the vendor had, and what we had before worked with our old system."
Top is terrible tool for measuring CPU time. Set up a bunch of counters or timer around the execution of individual and collect statistics on that.
Each of your block should have a "should cost" that's based on number of arithmetic operations plus some overhead for looping and conditionals relative to the instruction set of the processor. Compare each blocks actual cost to the "should cost" and look for major discrepancies
If the total doesn't add up, it's often caused by cache misses, pipeline stalls, memory bottlenecks, interrupts, or some other service barging in. Again, proper instrumentation can shed some light on what's happening.