Loading a word from byte-addressable cache
What makes you think that? 

If you have a modern high-performance CPU with 256 bit = 32 byte vector registers, then the byte offset determines which group of 1, 2, 4, 8, 16 or 32 consecutive bytes to transfer. Nobody in their right mind would transfer individual bytes. Much too slow. 

What will happen and may slow down things a little bit is that if the group of bytes to transfer is not contained in one single cache line, then two accesses are made to two different cache lines. For example, if your cachle lines are 64 bytes and you try to access a 32 bit (four byte) item at byte offset 61, 62 or 63.