Old switch has four wires and no discernable ground
I think I see what is happening.  Many devices (switches, outlets) allow you to connect via either the screw terminals or the "backstabs" as we call them (with some contempt because they tend to fail a lot). 

You're supposed to use only screws or backstabs.  But the screw is connected to the backstab internally, so this is a possible way to use both, which splices the  two wires and also connects to the device.  

It's just a lazy cheatey way to splice both those wires to the same screw without using a wire nut and pigtail.  

Now, I think this is a 3-way switch, but you didn't mention it being controlled in a second location, and the wiring doesn't make sense for a 3-way, so I think this is being used as a 1-way switch simply by not using one of the brass screws. 

The two yellow wires are definitely using the "use the switch as a splice" trick.  Both those wires should go to a wirenut, with a third short pigtail going to that same switch's screw.  Since your lights are in three locations, my guess is the two yellows are both switched-hot running in different directions out to the lamps.  

I can't quite tell if the black wires are on the same "screw", but it looks that way. Note how the 

It's a little harder to tell whether the blacks are on the same "screw"... But you see the stabbed black is a pigtail to a wire-nut. That is surely supply hot being distributed to several switches. Which would make the screw trick simply extending that hot further. If so, the wire on the screw now should go into that wire-nut, and the pigtailed wire should use the screw.

If it works out as I describe, now you have 2 wires going to that switch: the black is supply/always-hot/line, the yellow is switched-hot/load.  If your smart switch needs neutral, you can add a white pigtail to that bundle of neutrals in the back of the box.  (which you should not otherwise be messing with!)