Garden building (UK) 2.5m rule where measured?
Based on the info in your question;

If the structure is within 2m of the boundary then the height  of that structure is restricted to 2.5m.

That would mean that if your structure is within the 2m of the boundary then you can not have a structure  that is 2.75m in height   as it would violate the code.

Is that 2.5m measured from the immediate boundary or the actual
building itself?

The 2.5m height restriction is measured from grade (the ground), to the the top of the structure. Grade would be the ground at the bottom of the fence, not from the bottom of the hole you dug. Dig the hole deeper.
Again, this is based on how your question is worded.