Why Midoriya said he remembered All for One's memories?


In manga chapter #194

> After dreaming about former One for All owners and having some
> dreams/memories of the original One for All, Midoriya tells All Might about
> this and when replying to something All Might said, Midoriya says "Yeah.. it
> was also technically All For One's memories.

Why Midoriya said he remembered All for One's memories? I understand him
having One for All former owners memories, but All for One's ??

