Is a computer mouse considered an embedded system?
An embedded system is a computer which is found inside another device to serve a specific purpose. Old mice (with balls) did not have all the components of a computer inside them, so they were equipped with logic devices but not a computer and if you don't have a computer it can't be an embedded system.

The definition of a computer I'm using is of a device that can be programmed with some form of CPU, memory and perhaps some I/O. Other definitions exist and are more or less similar.

Modern optical mice have a processor integrated into an ASIC that drives a camera that takes pictures of a surface, records them to memory, and compares them against newer pictures and decides in which direction the device was moved. This qualifies as an embedded system so a modern mouse is an embedded system.

The fact you connect it as a peripheral is not contradictory.