Docker swarm custom scheduler
The scheduler itself is not swappable. However, you can set memory constraints and reservations on containers in your service:

version: '3'
services:
  redis:
    image: redis:alpine
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 50M
        reservations:
          cpus: '0.25'
          memory: 20M


The constraint will kill the container if it exceeds the limit even if the host has more memory available. The reservation will ensure that amount of memory on the host is available for the container.