Fields of math needed for digital filter design
To get started:

Complex numbers

The frequency response of a filter is easier to understand complex-valued, describing both the magnitude frequency response and the phase frequency response. You will be able to understand poles and zeros, which can be complex. Complex numbers enable you to have negative frequencies, which will make math simpler.

Trigonometry

$\sin$, $\cos$ and their relation to the complex exponential $e^{i\alpha} = \cos(\alpha) + i\sin(\alpha)$ are important. Sinusoidal functions will be passed through filters with only their amplitude and phase affected.

Differentiation

To find at what frequency a simple filter peaks or dips, you can solve at what frequency the derivative of its magnitude frequency response is zero.

Integration

Integration is needed for the Fourier transform and the inverse Fourier transform.

Fourier transform

Fourier transform enables you to go from an impulse response to a frequency response and back. Also things you do in the time domain often have a simple counterpart in the frequency domain, and vice versa.