<p dir="auto">Hello all - apologies if I'm creating a duplicate ticket...I haven't been able to conjure exactly what I'm looking for; this may be related to the issues described here: <a href="https://groups.google.com/forum/#!topic/google-containers/yg539XUMQSA" rel="nofollow">https://groups.google.com/forum/#!topic/google-containers/yg539XUMQSA</a>. I have tried going through the services walkthrough, to no avail.</p>
<p dir="auto">I am creating authenticated Mongo replica sets using Kubernetes on GCE. We are using custom, minimal images to run our software - I have tried the official images but have had the same problems with those.</p>
<p dir="auto">I can get Mongo to run without a hitch - the pods talk to each other, failed primaries trigger an election, syncing of documents occurs just fine.</p>
<p dir="auto">My troubles come in the form of the load balanced IP address assigned to these instances. Connections to the pods via the LB address are very unreliable, and recently have ceased to function at all. Connecting to the whole replica set is impossible.</p>
<p dir="auto">Our implementation does the following:</p>
<ul dir="auto">
<li>creates a persistent disk for the instance if it does not exist (gcloud compute disks create ...),</li>
<li>creates a PV/PVC binding to the disk,</li>
<li>creates a Service for each instance (with type: LoadBalancer),</li>
<li>creates a replication controller for each Mongo instance.</li>
</ul>
<p dir="auto">We're having to make a lot of these, so I'm using a pidgin template to generate all of these at runtime. In any event, all pods and services come up, with each service binding to the correct pod. Case in point:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ kubectl get pods | grep api-mongo
api-mongo-1-w358n   1/1       Running   0          1m
api-mongo-2-6j048   1/1       Running   0          59s
api-mongo-3-f8p0j   1/1       Running   0          57s

$ kubectl get svc | grep api-mongo
api-mongo-1        10.0.184.8     104.196.2.106     41572/TCP                    name=api-mongo-1,tier=data,track=stable                         1m
api-mongo-2        10.0.27.185    104.196.36.137    41572/TCP                    name=api-mongo-2,tier=data,track=stable                         1m
api-mongo-3        10.0.63.141    104.196.30.9      41572/TCP                    name=api-mongo-3,tier=data,track=stable                         1m

$ kubectl get pods --selector name=api-mongo-1,tier=data,track=stable
NAME                    READY     STATUS    RESTARTS   AGE
api-mongo-1-w358n   1/1       Running   0          2m

$ kubectl get pods api-mongo-1-w358n -o json  | grep podIP
        &quot;podIP&quot;: &quot;10.244.6.8&quot;,

$ kubectl get endpoints api-mongo-1
NAME              ENDPOINTS          AGE
api-mongo-1   10.244.6.8:41572   4m"><pre class="notranslate">$ kubectl get pods <span class="pl-k">|</span> grep api-mongo
api-mongo-1-w358n   1/1       Running   0          1m
api-mongo-2-6j048   1/1       Running   0          59s
api-mongo-3-f8p0j   1/1       Running   0          57s

$ kubectl get svc <span class="pl-k">|</span> grep api-mongo
api-mongo-1        10.0.184.8     104.196.2.106     41572/TCP                    name=api-mongo-1,tier=data,track=stable                         1m
api-mongo-2        10.0.27.185    104.196.36.137    41572/TCP                    name=api-mongo-2,tier=data,track=stable                         1m
api-mongo-3        10.0.63.141    104.196.30.9      41572/TCP                    name=api-mongo-3,tier=data,track=stable                         1m

$ kubectl get pods --selector name=api-mongo-1,tier=data,track=stable
NAME                    READY     STATUS    RESTARTS   AGE
api-mongo-1-w358n   1/1       Running   0          2m

$ kubectl get pods api-mongo-1-w358n -o json  <span class="pl-k">|</span> grep podIP
        <span class="pl-s"><span class="pl-pds">"</span>podIP<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>10.244.6.8<span class="pl-pds">"</span></span>,

$ kubectl get endpoints api-mongo-1
NAME              ENDPOINTS          AGE
api-mongo-1   10.244.6.8:41572   4m</pre></div>
<p dir="auto">This is as I would expect. When I attempt to shell into to Mongo, however:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ mongo -u admin --host 104.196.2.106 -p **** --port 41572 --authenticationDatabase admin admin
MongoDB shell version: 3.2.0
connecting to: 104.196.2.106:41572/admin
2016-02-05T10:47:57.113-0500 W NETWORK  [thread1] Failed to connect to 104.196.2.106:41572 after 5000 milliseconds, giving up.
2016-02-05T10:47:57.113-0500 E QUERY    [thread1] Error: couldn't connect to server 104.196.2.106:41572, connection attempt failed :
connect@src/mongo/shell/mongo.js:226:14
@(connect):1:6

exception: connect failed"><pre class="notranslate">$ mongo -u admin --host 104.196.2.106 -p <span class="pl-k">****</span> --port 41572 --authenticationDatabase admin admin
MongoDB shell version: 3.2.0
connecting to: 104.196.2.106:41572/admin
2016-02-05T10:47:57.113-0500 W NETWORK  [thread1] Failed to connect to 104.196.2.106:41572 after 5000 milliseconds, giving up.
2016-02-05T10:47:57.113-0500 E QUERY    [thread1] Error: couldn<span class="pl-s"><span class="pl-pds">'</span>t connect to server 104.196.2.106:41572, connection attempt failed :</span>
<span class="pl-s">connect@src/mongo/shell/mongo.js:226:14</span>
<span class="pl-s">@(connect):1:6</span>
<span class="pl-s"></span>
<span class="pl-s">exception: connect failed</span></pre></div>
<p dir="auto">Yesterday, I was able to shell in every 5-6 attempts - today, it never happens.</p>
<p dir="auto">Here's how the configurations look when rendered:</p>
<p dir="auto">Service:</p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="kind: &quot;Service&quot;
metadata:
    name: &quot;api-mongo-1&quot;
    labels:
        name: &quot;api-mongo-1&quot;
        tier: &quot;data&quot;
        track: &quot;stable&quot;
spec:
    type: &quot;LoadBalancer&quot;
    ports:
        - name: &quot;mongodb&quot;
          port: 41572
          targetPort: 41572
    selector:
        name: &quot;api-mongo-1&quot;
        tier: &quot;data&quot;
        track: &quot;stable&quot;"><pre class="notranslate"><span class="pl-ent">kind</span>: <span class="pl-s"><span class="pl-pds">"</span>Service<span class="pl-pds">"</span></span>
<span class="pl-ent">metadata</span>:
    <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>api-mongo-1<span class="pl-pds">"</span></span>
    <span class="pl-ent">labels</span>:
        <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>api-mongo-1<span class="pl-pds">"</span></span>
        <span class="pl-ent">tier</span>: <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>
        <span class="pl-ent">track</span>: <span class="pl-s"><span class="pl-pds">"</span>stable<span class="pl-pds">"</span></span>
<span class="pl-ent">spec</span>:
    <span class="pl-ent">type</span>: <span class="pl-s"><span class="pl-pds">"</span>LoadBalancer<span class="pl-pds">"</span></span>
    <span class="pl-ent">ports</span>:
        - <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>mongodb<span class="pl-pds">"</span></span>
          <span class="pl-ent">port</span>: <span class="pl-c1">41572</span>
          <span class="pl-ent">targetPort</span>: <span class="pl-c1">41572</span>
    <span class="pl-ent">selector</span>:
        <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>api-mongo-1<span class="pl-pds">"</span></span>
        <span class="pl-ent">tier</span>: <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>
        <span class="pl-ent">track</span>: <span class="pl-s"><span class="pl-pds">"</span>stable<span class="pl-pds">"</span></span></pre></div>
<p dir="auto">Replication Controller (truncated of environment variables and volumes):</p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="apiVersion: &quot;v1&quot;
kind: &quot;ReplicationController&quot;
metadata:
  name: &quot;api-mongo-1&quot;
  labels:
    name: &quot;api-mongo-1&quot;
    tier: &quot;data&quot;
    track: &quot;stable&quot;
spec:
  replicas: 1
  selector:
    name: &quot;api-mongo-1&quot;
    tier: &quot;data&quot;
    track: &quot;stable&quot;
  template:
    metadata:
      labels:
        name: &quot;api-mongo-1&quot;
        tier: &quot;data&quot;
        track: &quot;stable&quot;
    spec:
      containers:
        - name: &quot;api-mongo-1&quot;
          image: &quot;mongo:3.2.0&quot;
          ports:
            - name: &quot;mongo&quot;
              containerPort: 41572"><pre class="notranslate"><span class="pl-ent">apiVersion</span>: <span class="pl-s"><span class="pl-pds">"</span>v1<span class="pl-pds">"</span></span>
<span class="pl-ent">kind</span>: <span class="pl-s"><span class="pl-pds">"</span>ReplicationController<span class="pl-pds">"</span></span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>api-mongo-1<span class="pl-pds">"</span></span>
  <span class="pl-ent">labels</span>:
    <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>api-mongo-1<span class="pl-pds">"</span></span>
    <span class="pl-ent">tier</span>: <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>
    <span class="pl-ent">track</span>: <span class="pl-s"><span class="pl-pds">"</span>stable<span class="pl-pds">"</span></span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">replicas</span>: <span class="pl-c1">1</span>
  <span class="pl-ent">selector</span>:
    <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>api-mongo-1<span class="pl-pds">"</span></span>
    <span class="pl-ent">tier</span>: <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>
    <span class="pl-ent">track</span>: <span class="pl-s"><span class="pl-pds">"</span>stable<span class="pl-pds">"</span></span>
  <span class="pl-ent">template</span>:
    <span class="pl-ent">metadata</span>:
      <span class="pl-ent">labels</span>:
        <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>api-mongo-1<span class="pl-pds">"</span></span>
        <span class="pl-ent">tier</span>: <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>
        <span class="pl-ent">track</span>: <span class="pl-s"><span class="pl-pds">"</span>stable<span class="pl-pds">"</span></span>
    <span class="pl-ent">spec</span>:
      <span class="pl-ent">containers</span>:
        - <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>api-mongo-1<span class="pl-pds">"</span></span>
          <span class="pl-ent">image</span>: <span class="pl-s"><span class="pl-pds">"</span>mongo:3.2.0<span class="pl-pds">"</span></span>
          <span class="pl-ent">ports</span>:
            - <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">"</span>mongo<span class="pl-pds">"</span></span>
              <span class="pl-ent">containerPort</span>: <span class="pl-c1">41572</span></pre></div>
<p dir="auto"><code class="notranslate">iptables-save</code> on the node:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# iptables-save | grep api-mongo-1
-A KUBE-NODEPORT-CONTAINER -p tcp -m comment --comment &quot;default/api-mongo-1:mongodb&quot; -m tcp --dport 32113 -j REDIRECT --to-ports 45347
-A KUBE-NODEPORT-HOST -p tcp -m comment --comment &quot;default/api-mongo-1:mongodb&quot; -m tcp --dport 32113 -j DNAT --to-destination 10.240.0.19:45347
-A KUBE-PORTALS-CONTAINER -d 10.0.184.8/32 -p tcp -m comment --comment &quot;default/api-mongo-1:mongodb&quot; -m tcp --dport 41572 -j REDIRECT --to-ports 45347
-A KUBE-PORTALS-CONTAINER -d 104.196.2.106/32 -p tcp -m comment --comment &quot;default/api-mongo-1:mongodb&quot; -m tcp --dport 41572 -j REDIRECT --to-ports 45347
-A KUBE-PORTALS-HOST -d 10.0.184.8/32 -p tcp -m comment --comment &quot;default/api-mongo-1:mongodb&quot; -m tcp --dport 41572 -j DNAT --to-destination 10.240.0.19:45347
-A KUBE-PORTALS-HOST -d 104.196.2.106/32 -p tcp -m comment --comment &quot;default/api-mongo-1:mongodb&quot; -m tcp --dport 41572 -j DNAT --to-destination 10.240.0.19:45347"><pre class="notranslate"><span class="pl-c"><span class="pl-c">#</span> iptables-save | grep api-mongo-1</span>
-A KUBE-NODEPORT-CONTAINER -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/api-mongo-1:mongodb<span class="pl-pds">"</span></span> -m tcp --dport 32113 -j REDIRECT --to-ports 45347
-A KUBE-NODEPORT-HOST -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/api-mongo-1:mongodb<span class="pl-pds">"</span></span> -m tcp --dport 32113 -j DNAT --to-destination 10.240.0.19:45347
-A KUBE-PORTALS-CONTAINER -d 10.0.184.8/32 -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/api-mongo-1:mongodb<span class="pl-pds">"</span></span> -m tcp --dport 41572 -j REDIRECT --to-ports 45347
-A KUBE-PORTALS-CONTAINER -d 104.196.2.106/32 -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/api-mongo-1:mongodb<span class="pl-pds">"</span></span> -m tcp --dport 41572 -j REDIRECT --to-ports 45347
-A KUBE-PORTALS-HOST -d 10.0.184.8/32 -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/api-mongo-1:mongodb<span class="pl-pds">"</span></span> -m tcp --dport 41572 -j DNAT --to-destination 10.240.0.19:45347
-A KUBE-PORTALS-HOST -d 104.196.2.106/32 -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/api-mongo-1:mongodb<span class="pl-pds">"</span></span> -m tcp --dport 41572 -j DNAT --to-destination 10.240.0.19:45347</pre></div>
<p dir="auto">Mongo responds correctly to cURL requests from within the node, whether I'm making the request against the LB IP, or directly to the kube proxy on the node.</p>
<p dir="auto">Does anyone have any ideas about what I'm doing wrong here?</p>
<p dir="auto">Please let me know if I can provide any further information here. Thanks in advance!</p>
<p dir="auto">EDIT 1: edited containerPort on the RC - I fired my "dry run" code incorrectly when I posted this, and it rendered a default value in my template.</p>
<p dir="auto">The <a href="https://github.com/kubernetes/kubernetes/blob/master/test/e2e/configmap.go">configmap tests</a> should run as part of the node e2e suite. For now the tests can be duplicated in the <code class="notranslate">test/e2e_node</code> directory.</p>
<p dir="auto">Reserved for <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mnshaw/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mnshaw">@mnshaw</a></p>