How can I tell when to take a mulligan in Magic?
Unfortunately, knowing your deck inside out and comparing your draw against it are the only universal rules that can apply, so it's a highly individual decision that depends on those two factors. If you already have a good idea of the opponent's deck, that also helps, of course. Essentially, experience is where it's at.

You can ask yourself a few questions.


How many more spells/lands do I need to draw in succession to get things going as planned?
How badly is my strategy damaged if I fail to draw more of what I need?
In case of land shortage (mana screw), how many of my spells can I play with the cards I currently hold?
In case of excess lands (mana flood), how powerful are the spells I hold? 
Do the cards I 'can' play help me getting more spells by themselves (card drawers)?
Or are they strong enough to put pressure on my opponent, so I can buy a few turns to benefit from the regular card draw phase?
In multi-color decks and holding the wrong mana (color screw), how many colored mana am I missing to cast the majority of my hand, or how many more spells of the wrong color could follow?
What is the strategy of my opponent's deck? If he wins by putting out many fast creatures or direct damage, then losing a few turns may well mean defeat. If your opponent is playing a combo deck, all you may need is a single counter-spell that you can play at the right time.


Note that if you often get into a situation where you must consider mulligan (assuming careful shuffling before each game), then there might be an issue with your deck. Adjust the number of mana sources and/or the mana curve of your spells.

Shuffle thoroughly to ensure that the deck is randomised. Shuffling helps to ensure that if clumps of land or spells do occur, they do so naturally. Pile shuffling alone is not an acceptable means of shuffling. Mash shuffling and riffle shuffling are acceptable.

Mulligans are decided upon in APNAP order. They are acted upon in parallel. First, the active player chooses whether or not to mulligan. Then, the non-active player does the same. After both players have made a decision, mulligans are taken in parallel. This repeats until one player chooses not to mulligan. Then, the remaining player takes his or her remaining mulligans alone. 

Some important piece of information is born from this process. When the non-active player chooses to mulligan, the active player should have already stated his or her choice. This extra information may influence the non active player's decision to mulligan. For example, a hand full of one for one spells may be acceptable to keep against a player who is taking a mulligan, because one for one trades generally favor the player with more cards.