Need understanding of the concept of word (memory)
Your interpretation is correct.

In terms of the line:

"the total is 4 words x 4 bytes = 16 bytes which is 128 bits"

That is the total size of the memory. Each word in that memory is 32 bits, and you have 4 of them giving a total size of 128 bits.