Shimano 105 rear derailleur issue
It sounds like you are having trouble downshifting from any larger gear to a smaller one.
If a rear derailleur won't go from a larger gear to a smaller gear it's because:

The cable is binding preventing the derailleur from getting enough slack
The derailleur spring is too weak

Starting from a smaller gear on a 105 rear derailleur and moving to a larger gear happens because of tension on the cable. You move the lever and it pulls the cable which moves the derailleur. As the derailleur moves a spring in the derailleur is compressed storing energy.
When you move the lever to shift from a larger gear to a smaller gear the cable is given slack. The rear derailleur spring takes up the slack and moves to the smaller gear.
Get the bike on the rack or into a position where you can pedal and make the rear wheel move.
You can test to see if the cable is binding by:

Disconnecting the cable from the derailleur
Hold the just disconnected end in one hand and move the shift lever with the other. The cable should move easily in both directions.

If you feel any binding you'll need to hunt down the source and fix it.
It could be due to dirt/rust in the housing, a kink in the housing, a kink or hard bend in the cable etc.
With the cable disconnected you can test the derailleur spring by:

Turning the pedals with the rear wheel spinning
With the rear wheel moving use your hand to move the derailleur to shift into the larger gear.
Release the derailleur - keep pedaling - the spring in the derailleur should crisply shift back into the smaller gear.

If the derailleur won't move the chain from the larger gear to the smaller gear the spring may be weak or broken.