Create GUI components with states
Instead of comparing the class with if( source.getClass() == ColorButton.class) you can use if (source instanceof ColorButton). This is preferable in case you would make a subclass of ColorButton. If it's an instance of a subclass of ColorButton it is still an instanceof but source.getClass() != ColorButton.class.

With the way that you are using the code above, I would save the color in the listener instead of in the button. This way you don't need the if statement in your code at all, since you can pass the appropriate type to the Listener. In this case, .setBackground is declared in the JComponent class so all you need to make sure is that it's a JComponent that gets passed to the Listener, which is no problem:

static class Listener implements ActionListener {
    private Color color;
    private JComponent comp;
    public Listener(JComponent comp, Color color) {
        this.color = color;
        this.comp = comp;
    }
    @Override
    public void actionPerformed(ActionEvent actionEvent) {
        comp.setBackground(this.color);
    }
}


By using this, you don't need the ColorButton subclass, instead you create multiple instances of your listener:

JPanel panel = new JPanel();
JButton b = new JButton("Blue");
JButton r = new JButton("Red");
r.addActionListener(new Listener(panel, Color.BLUE));
b.addActionListener(new Listener(panel, Color.RED));
panel.add(b);
panel.add(r);
add(panel);