CPU Intensive task on Big Sur is really fast for 5 minutes, then really slow
We can dig deeper to understand what’s consuming the time, but let me preface this by saying that this may be normal behavior. I’ve seen plenty of cases where a compilation or linking phase takes a long time with large or complex projects.
Given that you’re not seeing thermal pressure and CPU utilization goes down to 10% during the slow phase, however, I’m inclined to suspect that your build is getting serialized behind a blocked process.
We can start to understand what process(es) is/are blocked by taking a stackshot during build:

Run the following Terminal command to enable kernel symbolication: $ sudo nvram boot-args=&quot;keepsyms=1&quot;
Reboot your machine and don’t open any apps except for Terminal.
Start your build and wait until you’ve entered the slow phase.
Run the following command, making sure you’re only running it during the window of time that the build is slow: $ sudo spindump -reveal -noProcessingWhileSampling
After waiting for 10 seconds to collect a sample and another minute or two to symbolicate and format, you’ll get a file in /tmp/spindump.txt that contains a stackshot of every process.
Upload the file to PasteBin or some equivalent place.

That will be the most direct way to begin root-causing. Also, just to rule out memory pressure, does your system “feel” sluggish to you during the build? (I suspect not, but you should run top in another Terminal window while you’re building anyway and monitor your swapouts to see if they’re growing.)
Without knowing any of the details of the code or its build process, a crude guess would be that it’s waiting on a resource (e.g. a network server that’s slow to resolve, or an indirect filesystem lookup on a slow partition/disk). If you feel like experimenting, try disabling all of your network interfaces, unmounting every partition, and disconnecting every device.