So my 3PAR SP has been messed up since my upgrade to 3.2.1 MU5. I've got a case open with HPE, but things aren't going well there. Initially, I was able to at least connect to the 3PAR with the management console. I ended up re-imaging the SP, but during the MOB, at the very end of the process, it cannot connect to the 3PAR with some sort of certificate error.
After going around and around with support, last week, one of the last thing HPE support had me do was a 'setsysmgr clearcache' on the array. I still could not get the SP to connect, so they want to dispatch a field engineer. I did nothing more and was out of the office between Thursday and yesterday.
This morning, I tried to connect with the management console, and it won't connect, just like the SP. I tried to connect with CLI, and it never connects. The only thing is, I can still connect to the array with SSH. I've been able to connect to it with SSH all along. I don't understand why the SP, CLI, and management console cannot connect. It wasn't clear what the setsysmgr clearcache command did, but it almost seems to have made things even worse.
Does anyone know why I'd be able to connect via SSH and suddenly, not through the SP, CLI, or management console?
So my last guess would be some sort of stuck process. SP <-> 3PAR exchange some sort of certificates after initial logon with 3paradm to ensure logins even after password changes. SSMC <-> 3PAR also exchange some certificates but that would be visible as an error when logging on to the SSMC admin interface.
CLI is stupid and the only thing I can think of why this doesn't work while SSH is would be if the CLI version is old or you're using something other than standard credentials when logging on.
SSMC - Check the admin console to see if a useful error, might be cert or just login needed. I've had to reboot/restart the SSMC server/service in the past after 3PAR upgrade before it would connect again (might have been a but in earlier SSMC version). Might have to remove the array and reboot SSMC and add again.
CLI - Check it's a matching version to the 3PAR OS, again maybe cert related so look for a folder (I can't remember where) which the CLI keeps certs in and delete the cert for that array.
SSH is what I use rather then CLI, the comma separated table output option is about the only thing I use CLI for as getting stuff out of SSMC is crap compared to the old GUI.
Not sure about the SP issues, not had problems from that end in the past, my guess is that the command you used deleted the SSL cert cache on the array so that it would request/accept new ones from previously known devices. Maybe there was a change in the format of the cert store that invalidated old cached copies.
The problem originated with the SP after the OS install finished. I upgraded the SP first to the compatible version, then the OS. The SP lost connectivity after the OS install. I figured I'd remove and re-add the array on the SP to re-establish communication. It was giving the key exchange error at that time. HPE Support has had me re-image the SP a bunch of times, to no avail. Now they're dispatching someone to come on-site to re-image the SP, which I've already done.
I really feel like the problem is on the array because it's not allowing the SP to communicate with it.
Also surpriced they don't try to redeploy SP remote. I see this resulting in the problem not fixed and the guy coming onsite is only there for the SP, not looking at the 3PAR.
Exactly, which is why I'd love to fix this myself before he gets here. I've already re-deployed the SP multiple times, both physical AND virtual. It can't pass the validation at the end of MOB when it tries to connect with the array. I just can't figure it out. The last thing one of the engineers on the phone had me run was a 'setsysmgr clearcache' command. I think things got even worse after that.
I've tried to recreate the self-signed cert, but that didn't help any. There's something on the array that's "stuck" or something. I've rebooted both nodes separately.
We're a dark site to HPE, so they can't remotely re-deploy the SP. I suspect they'd run into the same issue though, with the number of tries I've made at it.
Yeah it might take a while with no direct support link, I think we didn't go beyond MU3 on 3.2.1 before going to 3.2.2 but I had a look at the MU5 release notes and it does seem to replace a lot of the SSL/SSH packages so wouldn't be surprised if something linked to that caused an issue. Maybe double check all your package versions against the expected list in the release notes. Any post MU5 updates related to network side?
Also out of interest as it's 3.2.1 does the old IMC GUI fail too?
We're at 3.2.1 MU5. I've tried the old IMC as well. That also will not connect.
Out of curiosity, does anyone know - when you run through the MOB when you re-deploy the SP, it gives you the option to load a backed up key/cert from a previous version. We have four arrays and never once has anyone asked us to backup a key/cert for anything. Is this cert the same self-signed cert from the array when you run 'showcert'?
I've not built an SP directly but have watched 3PAR/HP/HPE engineers do it when we've had SPs swapped out or the boot disk replaced. I don't remember them bothering to copy anything (disk being dead obviously not possible) to the new disk, normally a fresh install and type in all the details again with same IP address. It's been a while but I think they managed to do it all from the SP console, with no need to direct connect to the array.
But obviously as you have more stuff not connecting then the SP it looks more of an array issue (assuming nothing blocking ports at the network side as was working before upgrade).