The meaning of Bandwidth
Bandwidth has the same definition in control theory as in filters and other systems. It is usually defined as the point where the amplitude drops by 3 decibels (-3 dB) which is a factor of \$1/\sqrt{2} \approx 1/1.414\$. The reason that the factor \$\sqrt{2}\$ was chosen has to do with that it aligns with the "frequency" of a pole or zero. In other words, for a single real pole in the left-half plane \$p\$:

$$H(j\omega) = \frac{1}{1 + \frac{j\omega}{p}}$$

At the frequency where \$|\omega| = p\$, the amplitude of this system will be \$\frac{1}{1 + 1j}\$ which has an amplitude of \$1/\sqrt{2}\$ (hence -3dB).

Stability is more a complex issue and depends on multiple factors. The typical way to analyze it is by looking at the open-loop transfer function to determine what will happen by closing it. Bandwidth has an impact, but is definitely not the only factor that determines stability. The most common ways of analyzing stability are:


Bode's stability criterion, which is unfortunately not a necessary condition for stability, but it is sufficient. It is simple to use and works in a lot of cases.
Nyquist's stability criterion is harder to understand and analyze, but it is a necessary and sufficient condition.


I'll not go deeper into these topics, as your question was


  What is the relationship between Bandwidth of a system and stability of a system?


There is no clear relationship between the system bandwidth and stability. There are some cases where you can find a relationship that includes the bandwidth though.

In some practical cases, the poles of the open-loop system are all on the real axis of the left-half complex plane, and there aren't any zeroes in the system. We can then distinguish a dominant pole at a frequency \$f_d\$ which is the lowest frequency pole, and a non-dominant pole at a frequency \$f_{nd}\$ which is the second-lowest frequency pole.

Each pole will introduce a phase shift of \$90^\circ\$, so after two poles, the total will be \$180^\circ\$. At this frequency, Bode tells us that the amplitude of the open-loop system should not be at 0 dB or higher, or else the system will certainly be unstable. This means that we would want at least that \$f_{nd} &gt; f_{unity\ gain}\$ where \$f_{unity\ gain}\$ si the frequency where the open-loop system crosses 0 dB. This unity gain frequency can often be approximated by the so-called Gain-Bandwidth (GBW),

$$GBW = A_{DC}\cdot BW$$

If \$f_{nd} = A_{DC}\cdot BW\$ then the system will probably be unstable, or the system may be stable but only barely (it will still oscillate and that oscillation will only die out very slowly). Typically, you at least want to have something along the lines of

$$f_{nd} &gt; k\cdot A_{DC}\cdot BW$$

with \$k\$ a factor 3 to 4 depending on how stable you want your system.

Once you add zeroes, or if your poles are not on the real axis, the above reasoning becomes more complex. Bode and Nyquist will help you in that case, but there may not be a clear-cut relationship between the bandwidth and stability.

As far as I can tell, the closed-loop system bandwidth does not have anything significant to do with stability. The open-loop system will define the closed-loop bandwidth and stability.