So I spent the last 2 weeks dealing with ridiculous Media Resource Issues after our upgrade. The problem was always the same… Mobile Agent gets a call and answers it to dead air. The issue is intermittent and happening at multiple sites. At first we think it has something to do with the 9.1.2. 12033-1 ES we have just installed on our CUCM cluster that is having the most issues in order to address other media resources hanging on routers after the upgrade to 9.1.2(SU1). That bug is CSCum05362 and we are still resetting resources on every router on our other 2 clusters every night to prevent disaster.
Another thing first, in our old 7.x CVP/CUCM we were using mixed codec with our agents even though it wasn’t technically supported. In the new environment it is now supported and wasn’t working very well. All normal hard phones seemed to be working OK, but mobile agents doing transfers were really struggling. Every site with any Mobile Agents is now G.711 all the way through to get us back functioning after the upgrade.
Now switching the entire call flows to G.711 at the mobile agents sites helped, but we were still getting random problems with agents reporting dead air, busy signals and call drops.
This obviously sounds like media resources to me, so I go to the most obvious place to find Media Resource Problems, RTMT. Now this is super annoying, RTMT has 0 MediaResourceListExhausted messages related to the sites with the issues. We went back and forth with the business and taking random guesses at fixes when issues would come up, but would eventually attribute our fix to more specific MRG settings. We are also looking at the CUSP logs and for some reason the failed calls aren’t showing up in either. This is very odd.
The first big break we got was from CVP Call Server Logs (with debug 41 turned on)… You can find these debug settings at http://localhost:8000/cvp/diag on the call server. This is the magic interface TAC use’s they never tell you about in the guides.
We go open up C:\Cisco\CVP\logs\Error-<DATE> and look around a bit and start seeing these guys coming up every few minutes… FINALLY OUR MISSING FAILED CALLS EXIST!
At this point its pretty clear we have a media resource issue, but it’s intermittent, the CUCM gives us no RTMT errors and in our particular setup we have hard MTP (G.711 and G.729) we have Transcoder up the yazoo and enough CFB to host a Facebook earning call.
This is where I decide to break out the CUCM traces, since I determine there has to be a resource problem regardless what CUCM says. I go take a look at the SUB our problem phone is registered too and go pull logs.
Now I personally am seeing something like this in the logs….
I have no idea how to read this and didn’t until after I fixed the problem however I can clearly see its MTP related. Now we have had so many MTP issues over the years we finally broke down and configured software sessions on each router for g729 AND g711 so we would NEVER run out of sessions. Also in CUCM 7.x the software MTP were pretty unreliable.
I go monitor the resources in RTMT and indeed they aren’t getting used when we test, but Software MTP on the CUCM are getting used. This is completely baffling because in our MRGL its something like the following. Please note MTP are on top of Transcoders because the transcoders will get used for an MTP instead if they are on top. We are sure they were getting used in the old 7.5 cluster.
Turns out it’s dying intermittently when all the Software MTP’s are used up and there is none left to allocate. This is starting to make some sense… other then why my hardware MTP’s aren’t being used.
Here is where I find my golden ticket to the solution. Straight from an older 7.5 SRND.
The odd part of this is that these are incoming calls to agents and have no need for DTMF but this has to be it. I decided its gamble time, let’s add pass-through to the mtp profiles.
A few test calls later and RTMT confirming we have the issue fixed, we are using the hardware MTPs. A show sccp connections command on the ISR with the MTP’s configured will show them actively in use.
Long in short this is a great blog and goes through all the steps I did and comes to the ultimate conclusion this is a new feature!
This one still leaves a lot to be figured out. CUSP should show failed calls and RTMT surely should alert on having no resources available. The other linked blog also contains some great information on the CAP Codes in the MTP traces. I just wish I would have found that blog 10 days early and saved myself about 60 hours of troubleshooting and tail chasing.
Thanks for this – when I saw it all the rummaging through the MRGL and the scup traffic to the IOS MTPs were enough to send CUCM into code yellow.