Understanding Bernoulli Trials, Bayesian Setting
What is a Bayesian setting and why is Bayesian thing everywhere?


In very simple terms:


  Bayesian is a statistical setting, where the likelihood of an event
  happening (called the posterior) depends on the prior trials or
  observations (called the prior(s)).


Bayesian networks is an extension of the above, forming a chain or a network of inferencing.


  Are there any theorems/theories that I must know?


For understanding the Bayesian paradigm, you need to know the Bayesian theorem/relation, which is basically:

$$P(\theta|d) = \dfrac{P(d|\theta)P(\theta)}{P(d)}.$$


  Any book/notes/resources where I could learn about these stuff in a
  relatable way (I have very little maths background, but I can learn)?


I would highly recommend "Doing Bayesian Analysis" by John Krushke