title: Accessing UI for spark running as kubernetics container on standby name-node
body: <p>We are launching all spark jobs as kubernetics(k8es) containers inside a k8es cluster. We also create a service on each job and do port forwarding for the spark UI (container's 4040 is mapped to SvcPort, say, 31123).
  The same set of nodes is also hosting a Yarn cluster.
  Container's entry command is a call to spark-submit to Yarn in client mode.</p>
  
  <p>Now, the spark container may get launched on any of the name-nodes - master or standby. There is a VIP assigned to the active name-node.</p>
  
  <p>When spark container gets launched on active name-node, spark UI and all its tabs are easily accessible from anywhere using VIP:SvcPort
  However, when spark container gets launched on standby name node, spark UI is NOT accessible and eventually request fails with 500 error (as the VIP:SvcPort gets redirected to ActiveNamenode:8080 which is not accessible as driver is now running on standby name node.</p>
  
  <p>I have tried to reason and try out multiple service configurations but nothing seems to work this scenario. This config is what I have tried using which UI is accessible when container is on master node but UI is NOT accessible when container is on standby node.</p>
  
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
     name: spark-svc
     labels:
       deployer: sample_app
       app: spark-app
  spec:
    selector:
      app: spark-app
    ports:
    - port: 31123
      protocol: TCP
      targetPort: 4040
      name: ui
    externalIPs:
    - 192.168.122.22
  </code></pre>
  
  <p>May anyone pls suggest proper k8es service config for this scenario. We do not want to restrict the spark containers to master name-node as let k8es address the job load within k8es cluster.</p>
  
  <p>Any help is much appreciated.</p>