How do I fix this part of the wall?
It looks like you scrubbed away the top layer of paint. There's no way to scrub it back on. 

If you have the paint, you can easily blend it in, though fading due to age may result in it being slightly conspicuous. 

If not, get a paint match (for both color and sheen) and paint the entire wall.