Selecting multiple checkboxes to show similar but different boxes
A few notes on your code:

If you don't need supporting of very old or non-standard browsers, and given that your project does not relies heavily on jQuery, vanilla JavaScript can do the same work, without jQuery's overhead.
On each checkbox click, all coloured divs are queried again and again (jQuery('.box')), which is unnecessary as there are no changes in the DOM.
All checkboxes are queried multiple times, instead of selecting them once.
The if-then structure is quite cumbersome: if #red is checked, show .red.box. Then, if #green is also checked, hide .red.box and show .redgreen.box. Then, if #blue is checked too, hide .redgreen.box and show .rbg.box. One way for improving this part would be to first determine exactly which box should be checked and show only that box.
Personally, I would have used combinations of the .red/.green/.blue classes, instead of putting a different class for each combination (e.g. .box.red.green.blue instead of .box.rgb). That way each checkbox corresponds with a particular div class.

Here is my take on the problem/homework/exercise, that addresses most of the above remarks (no CSS/HTML changes; still using jQuery):
        const RED = 1;
        const GREEN = 2;
        const BLUE = 4;

        const DIV_CLASSES_MAP = [null, '.red', '.green', '.redgreen', '.blue', '.redblue', '.greenblue', '.rgb'];
        const DIVS = $('.box');
        const RED_TICK = $('#red');
        const GREEN_TICK = $('#green');
        const BLUE_TICK = $('#blue');

        let activeDiv = DIVS.filter(':visible');
        $('input[name=&quot;colorCheckbox&quot;]').on('click', onCheckboxChange);

        function onCheckboxChange() {
            const redBit = RED_TICK.is(':checked') ? RED : 0;
            const greenBit = GREEN_TICK.is(':checked') ? GREEN : 0;
            const blueBit = BLUE_TICK.is(':checked') ? BLUE : 0;
            const mask = redBit + greenBit + blueBit;
            const activeClass = DIV_CLASSES_MAP[mask];
            activeDiv.hide();
            activeDiv = DIVS.filter(activeClass).show();
        }