MONGODB primary became secondary after servers restart I had this config before restart:

{
    "_id" : "rs0",
    "version" : 5,
    "members" : [
        {
            "_id" : 0,
            "host" : "Ubuntu12041vanilla:27017"
        },
        {
            "_id" : 1,
            "host" : "slave-ubumaster-9mnltsm2.srv.ravcloud.com:27017"
        },
        {
            "_id" : 2,
            "host" : "slave2-ubumaster-ndww0zab.srv.ravcloud.com:27017"
        }
    ]
}


first node was primary...
After the restart of all servers, all of them became secondaries.
So I am trying to push the first one back to be primary, but it is not working.

cfg = rs.conf()
cfg.members[0].priority = 0.5
cfg.members[1].priority = 2
cfg.members[2].priority = 2
rs.reconfig(cfg, {force:true})// the force is because I am secondary


So I am getting this error:

{
    "assertion" : "need most members up to reconfigure, not ok : slave2-ubumaster-ndww0zab.srv.ravcloud.com:27017",
    "assertionCode" : 13144,
    "errmsg" : "db assertion failure",
    "ok" : 0
}


And I understand why: this host (slave2-ubumaster-ndww0zab.srv.ravcloud.com:27017) is not functioning anymore because of a network problem.
So I tried to change the hosts: cfg.members[1].host = "slave:27017"
but I get this error:

{
    "assertion" : "assertion db/repl/rs_initiate.cpp:119",
    "errmsg" : "db assertion failure",
    "ok" : 0
}


So I am stuck in a loop where I cant bring up the primary and I cant change the hosts to fix the network problems.

Do you see any chance to get out of that loop before giving up the replica set?

rs.status()
{
    "set" : "rs0",
    "date" : ISODate("2014-02-14T16:34:00Z"),
    "myState" : 2,
    "members" : [
        {
            "_id" : 0,
            "name" : "Ubuntu12041vanilla:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "optime" : {
                "t" : 1392323075000,
                "i" : 1
            },
            "optimeDate" : ISODate("2014-02-13T20:24:35Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "slave-ubumaster-9mnltsm2.srv.ravcloud.com:27017",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : {
                "t" : 0,
                "i" : 0
            },
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : 0,
            "errmsg" : "socket exception"
        },
        {
            "_id" : 2,
            "name" : "slave2-ubumaster-ndww0zab.srv.ravcloud.com:27017",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : {
                "t" : 0,
                "i" : 0
            },
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : 0,
            "errmsg" : "socket exception"
        }
    ],
    "ok" : 1
}


Thanks
You need to change the config such that there is only one member remaining, and you have to do it on that host, i.e. Ubuntu12041vanilla:27017

The others are unreachable from its perspective, so even if you go with 2 nodes, 1 out of 2 down means you cannot form a majority and you will still only have secondary state.

Given your current rs.status() output, try this when connected via the mongo shell to Ubuntu12041vanilla:27017:

cfg = rs.conf()
cfg.members = [cfg.members[0]]
rs.reconfig(cfg, {force : true})


That will give you a config only one node and should mean that Ubuntu12041vanilla:27017 will be the only remaining member and primary.