title: Running Apache Beam python pipelines in Kubernetes
body: <p>This question might seem like a duplicate of <a href="https://stackoverflow.com/questions/57851158/how-do-i-run-beam-python-pipelines-using-flink-deployed-on-kubernetes">this</a>.</p>
  
  <p>I am trying to run Apache Beam python pipeline using flink on an offline instance of Kubernetes. However, since I have user code with external dependencies, I am using the Python SDK harness as an External Service - which is causing errors (described below).</p>
  
  <p>The kubernetes manifest I use to launch the beam python SDK:</p>
  
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: beam-sdk
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: beam
        component: python-beam-sdk
    template:
      metadata:
        labels:
          app: beam
          component: python-beam-sdk
      spec:
        hostNetwork: True
        containers:
        - name: python-beam-sdk
          image: apachebeam/python3.7_sdk:latest
          imagePullPolicy: "Never"
          command: ["/opt/apache/beam/boot", "--worker_pool"]
          ports:
          - containerPort: 50000
            name: yay
  
  apiVersion: v1
  kind: Service
  metadata:
    name: beam-python-service
  spec:
    type: NodePort
    ports:
    - name: yay
      port: 50000
      targetPort: 50000
    selector:
      app: beam
      component: python-beam-sdk
  </code></pre>
  
  <p>When I launch my pipeline with the following options:</p>
  
  <pre><code>beam_options = PipelineOptions([
      "--runner=FlinkRunner",
      "--flink_version=1.9",
      "--flink_master=10.101.28.28:8081",
      "--environment_type=EXTERNAL",
      "--environment_config=10.97.176.105:50000",
      "--setup_file=./setup.py"
  ])
  </code></pre>
  
  <p>I get the following error message (within the python sdk service):</p>
  
  <pre><code>NAME                                 READY   STATUS    RESTARTS   AGE
  beam-sdk-666779599c-w65g5            1/1     Running   1          4d20h
  flink-jobmanager-74d444cccf-m4g8k    1/1     Running   1          4d20h
  flink-taskmanager-5487cc9bc9-fsbts   1/1     Running   2          4d20h
  flink-taskmanager-5487cc9bc9-zmnv7   1/1     Running   2          4d20h
  (base) [~]$ sudo kubectl logs -f beam-sdk-666779599c-w65g5                                                                                                                   
  2020/02/26 07:56:44 Starting worker pool 1: python -m apache_beam.runners.worker.worker_pool_main --service_port=50000 --container_executable=/opt/apache/beam/boot
  Starting worker with command ['/opt/apache/beam/boot', '--id=1-1', '--logging_endpoint=localhost:39283', '--artifact_endpoint=localhost:41533', '--provision_endpoint=localhost:42233', '--control_endpoint=localhost:44977']
  2020/02/26 09:09:07 Initializing python harness: /opt/apache/beam/boot --id=1-1 --logging_endpoint=localhost:39283 --artifact_endpoint=localhost:41533 --provision_endpoint=localhost:42233 --control_endpoint=localhost:44977
  2020/02/26 09:11:07 Failed to obtain provisioning information: failed to dial server at localhost:42233
      caused by:
  context deadline exceeded
  </code></pre>
  
  <p>I have no idea what the logging-  or artifact endpoint (etc.) is. And by inspecting the source code it seems like that the endpoints has been hard-coded to be located at localhost.</p>