Dynamic Fibonacci algorithm in x86 (MASM)
I'm wondering how this is normally done


Yours is as good a solution as anyone else's.
That said, a solution that pushes the elements would not need to address these elements relative to EBP at all.




; new f[] of size n + 2 
lea     edi, [ebx + 2]             ; save size of array in edi
shl     edi, 2                     ; mul by 4, size of DWORD int



You can calculate the required stackspace in a single LEA instruction:

; new f[] of size n + 2 
lea     edi, [ebx * 4 + 8]





; f[0] = 0 and f[1] = 1         
mov     DWORD PTR[esp], 0
mov     DWORD PTR[esp + 1 * 4], 1                       
; for (i = 2; i &lt;=n; i++)
mov     ecx, 2                     ; loop count i set to 2



All of these assignments require a 32-bit immediate value. That's a lot of bytes! Because later we need ECX=2, we can start with clearing ECX and incrementing twice towards 2, storing the intermediate results in memory:

; f[0] = 0 and f[1] = 1         
xor     ecx, ecx
mov     [esp + 0 * 4], ecx         ; f[0] = 0
inc     ecx
mov     [esp + 1 * 4], ecx         ; f[1] = 1
inc     ecx                        ; loop count i set to 2


But wait, there still room for improvement!


The address [esp + 0 * 4] requires a modR/M-byte, sib-byte and an 8-bit displacement.
The address [esp + ecx * 4] requires a modR/M-byte and a sib-byte.


Final draft:

; f[0] = 0 and f[1] = 1         
xor     ecx, ecx
mov     [esp + ecx * 4], ecx       ; f[0] = 0
inc     ecx
mov     [esp + ecx * 4], ecx       ; f[1] = 1
inc     ecx                        ; loop count i set to 2





; f[i] = f[i - 1] + f[i - 2]
lea     eax, [ecx - 1]
mov     esi, DWORD PTR[esp + eax * 4]
dec     eax
add     esi, DWORD PTR[esp + eax * 4]
mov     [esp + ecx * 4], esi



You can easily address the 2 previous array elements by adding offsets -4 and -8.
This liberates the ESI register that also no longer needs to be preserved.

; f[i] = f[i - 1] + f[i - 2]
mov     eax, [esp + ecx * 4 - 8]   ; f[i - 2]
add     eax, [esp + ecx * 4 - 4]   ; f[i - 1]
mov     [esp + ecx * 4], eax       ; f[i]





        mov     ecx, 2
L1:     cmp     ecx, ebx
        jg      endL1
        ...
        inc     ecx        ; inc i
        jmp     L1
endL1:



A loop like this is wasteful because of the uncondional jump that gets executed on every iteration, on top of the conditional jump that needs to be  processed anyway. You can make the conditional jump the only jump that gets executed repeatedly.

        mov     ecx, 2
        jmps    L2         ; once
L1:     ...
        inc     ecx        ; inc i
L2:     cmp     ecx, ebx
        jbe     L1         ; repeatedly