What are the basic categories of operations performed on audio signals to create interesting sounds?
The AM / FM category is a bit strange for effects - true, you can get some interesting effects by modulating the amplitude of an input signal ; but what does "frequency modulation" mean for a complex input signal for which you don't even have an accurate frequency representation? You could very well say that every effect is an amplitude modulation, by the $\frac{out(t)}{in(t)}$ signal, but this is not going to help you implement it!

There is no magic single-purpose engine at the heart of all the audio effects provided by music production software. But if you look at the source code of a large suite of audio effects from a DSP developer, here are some "building blocks" that will have been factored out in their own classes and which are shared by many different effects. I don't imply that this forms a good basis to categorize effects ; but from an implementation point of view, those blocks are essential:


Biquad filters.
Fractional delay lines.
Convolution engine, for fast convolution of an input signal with arbitrary-sized impulse responses.
Waveshaper (application of a non-linear function to the input signal in the time-domaine).
Synthesizer building blocks: oscillators, LFOs, ADSR envelopes.
Signal detectors: envelope follower, f0 extractor.


With those blocks you could build:


Synth-like filters or wah-wah: one or two biquads
Auto-wah: envelope follower + envelope + biquad
Flanger / Phaser: LFO + delay line
Chorus: LFO + delay lines
Algorithmic reverb: delay lines (array of parallel and serial comb filters)
Convolution reverb: convolution
Spatializer: convolution (with HRTFs impulse responses)
Guitar amp simulation: convolution, waveshappers
Distortion, Fuzz: gain + waveshapper
Bitcrusher (quantizer): a particular case of waveshapper
Ring modulator with a sine carrier: oscillator
Noise Gate and other dynamics effects: envelope follower
Tape simulation: convolution, waveshapper, envelope follower (for the dynamics processing)
EQ: biquads
Tremolo: LFO
Leslie emulation: convolution + LFO
Vocoder: biquads + envelope follower


While some of these effects are just a matter of patching the different blocks (a flanger truly is a LFO modulating a delay line), some other effects need more complex glue between the parts, that is specific enough not to be factored into a building block of its own.

This is, overall, an interesting set of effects that will cover a lot of ground for music production, but it's not exhaustive and there are indeed some effects which do not fit this framework... Some examples:
* Dirty sample rate reducer (in a bitcrusher): true, it's a multiplication by a dirac comb and then a convolution by a rectangular window... but it's easier to write it as something that duplicate the value of one sample over the N following samples instead of patching an amplitude modulation and a convolution!
* Pitch alteration effects (pitch-shifting, auto-tune) do not fit this framework well. They need a more complex representation of the audio signal to be performed (phase vocoder for frequency-domain methods ; pitch detection and pitch marks for time-domain methods).

I suggest you to play with "modular style" software tools like Pd, Reaktor, Plogue, SyntheEdit... - and try implement effects from the basic building blocks they provide you.