Minimum mountain groupset hard enough to resist abuse
Note: this is mostly based on my personal experience. Its an
  incredibly subjective subject, so there is no right answer. Keep in
  mind that in reality one level up or down will be completely
  unnoticeable performance difference to most riders.


XT is considered the "Sweet spot" for performance, weight and durability.  Probably X9 in the SRAM range. XTR and X0 are for sponsored riders, rich competitive riders and those that have to be seen to have the best because they will never be...

Alivo is definitely entry level and although it starts out OK, is less durable and quickly becomes unreliable if used hard. Deore and X7 are known for good durability - not up to  XT in quality and feel and durability - but not far behind. IMHO Deore and X7 are a good price point for a budget conscious weekend warrior - especially if you can get last years stuff at a good price. 

So depending on your budget, I would recommend looking towards Deore  or SRAM X7 level gear. From you post, I suspect XT is too good for the bike and you won't get the advantage of the extra $$$.

So where is you problem? The bike "was good once" - so the level of the gear is not the problem. I suspect the gear on it is worn out..... 

I would start with a close look at the chain lines - check the Bottom bracket for movement, and check the derailleur hanger for alignment (LBS has a tool for this). Check the cables are moving freely - I suggest replace the rear derailleur if you have any doubt or its more than a year old. All this is low or no cost and can make an enormous difference if they are out. 

Assuming these are OK, I would start with a closer look at the chain, cluster and chain rings. Especially check the chain is clean and free running, with no tight links. If these have not been replaced recently, its the first thing to consider. At the same time, check the derailleur for movement - this is hard if you are not experienced. A little is OK, a lot not - and how to tell - got the LBS and look at some new bikes, look at some mates bikes. 

Here is where you can start throwing money into a seemingly bottom less pit, so be careful. Don't feel the need to replace with higher spec components; don't go up more than 1, at a stretch 2 levels - so Alivio to Deore is OK, Alivio to XT - think very carefully. Keep it all about the same. 

You are likely to find that parts prices mean an upgrade of bike is worth considering. A MTB is more than the drive line - tires, shocks, frame geometry all make a bigger difference than smooth gear changes.