Small, two-way linear actuator
So, at least I found out that what I am looking for exists. It is called a reversible (or double-action) solenoid actor *. Here is a link to an example product listing.

Unfortunately, all reversible solenoid actors I found on the market are rather big, and my idea behind using one was that I wanted something with a small form factor (smaller than e.g. a step motor or rotational actuator). It seems, if I ever get around to building my project, I would just place a magnet on the thing I want to push, and place a reversible electromagnet at some distance.



* I actually found it via the German term, Umkehrhubmagnet (in case it helps anybody).