title: GCP - Regional GKE cluster with Network endpoint group NEG (with HTTP LoadBalancer and Cloud Armor)
body: <p>thanks to Gabriel Hodoroaga and his <a href="https://stackoverflow.com/questions/72950423/gcp-external-http-cloud-load-balancer-with-nginx-ingress-on-gke/72959225#72959225">tutorial</a> we have config with this flow in GCP:</p>
  <blockquote>
  <p>Internet &gt; HTTP Load Balancer &gt; Network Endpoint Groups &gt; GKE in one zone &gt; ingress-nginx</p>
  </blockquote>
  <p>But we need to switch GKE from zonal to regional. So I rebuild this config but lot of thinks I did manually via gcloud commands.
  I believe there is some better solution because this have big disadvantages:</p>
  <ul>
  <li>It works only on the initial deployment. If the pod with ingress-nginx is later moved to different zone (after restart), the connection is interrupted and the backend must be manually re-associated with the correct neg.</li>
  <li>We need to apply it in our environment where we don't use gcloud commands but everything is set via github actions with terraform.</li>
  </ul>
  <p>After initial deployment it works well: <a href="https://i.stack.imgur.com/jVbbg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jVbbg.png" alt="enter image description here" /></a>
  But after restarting ingress-nginx pod it moves to different zone and backend stay attached to old one:<a href="https://i.stack.imgur.com/JwR7S.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JwR7S.png" alt="enter image description here" /></a></p>
  <h4>Our config is described in following tutorial:</h4>
  <p><a href="https://github.com/robinpecha/gcp-regionalgke-httplb-negs-ingressnginx/blob/main/lb-negs-nging-reg.sh.md" rel="nofollow noreferrer">https://github.com/robinpecha/gcp-regionalgke-httplb-negs-ingressnginx/blob/main/lb-negs-nging-reg.sh.md</a></p>
  <hr />
  <h2>GCP - HTTP LOAD BALANCER &gt; NEGS &gt; REGIONAL GKE CLUSTER &gt; INGRESS-NGINX</h2>
  <p>Based on <a href="https://gist.github.com/gabihodoroaga/1289122db3c5d4b6c59a43b8fd659496" rel="nofollow noreferrer">tutorial</a> of <a href="https://hodo.dev/posts/post-27-gcp-using-neg/" rel="nofollow noreferrer">Gabriel Hodoroaga</a>.</p>
  <h2>Vars</h2>
  <p>Replace at least YOURDOMAIN.</p>
  <pre><code>CLUSTER_NAME=&quot;lb-negs-nging-reg&quot;
  REGION=&quot;europe-west2&quot;
  YOURDOMAIN=&quot;put-your-domain.here&quot;
  echo $CLUSTER_NAME ; echo $REGION ; echo $YOURDOMAIN
  </code></pre>
  <h2>Create the cluster</h2>
  <pre><code>gcloud container clusters create $CLUSTER_NAME --region $REGION --machine-type &quot;e2-medium&quot; --enable-ip-alias --num-nodes=2 
  </code></pre>
  <h2>add the helm ingress-nginx</h2>
  <pre><code>helm repo update
  helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
  </code></pre>
  <h2>Install the ingress-nginx</h2>
  <p>Create a file values.regional.yaml for ingress-nginx:</p>
  <pre><code>cat &lt;&lt; EOF &gt; values.regional.yaml
  controller:
    service:
      type: ClusterIP
      annotations:
        cloud.google.com/neg: '{&quot;exposed_ports&quot;: {&quot;80&quot;:{&quot;name&quot;: &quot;ingress-nginx-80-neg&quot;}}}'
  EOF
  </code></pre>
  <p>And install it:</p>
  <pre><code>helm install -f values.regional.yaml ingress-nginx ingress-nginx/ingress-nginx
  </code></pre>
  <h2>install dummy web server</h2>
  <p>Prepare config:</p>
  <pre><code>cat &lt;&lt; EOF &gt; dummy-app-lightweb.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: lightweb
  spec:
    selector:
      matchLabels:
        app: dummy
    replicas: 3
    template:
      metadata:
        labels:
          app: dummy
      spec:
        containers:
        - name: lightweb
          image: alastairhm/alpine-lighttpd-php
          ports:
          - name: http
            containerPort: 80
          lifecycle:
            postStart:
              exec:
                command: [&quot;/bin/sh&quot;, &quot;-c&quot;, 'wget https://raw.githubusercontent.com/robinpecha/hello-world/main/php-header/index.php -P /var/www/']
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: dummy-service
  spec:
    type: NodePort
    ports:
    - port: 80
      targetPort: 80
    selector:
      app: dummy
  EOF
  </code></pre>
  <p>Apply this config:</p>
  <pre><code>kubectl apply -f dummy-app-lightweb.yaml 
  </code></pre>
  <p>Now you can check if is your dummy web server works :</p>
  <pre><code>kubectl get pods
  #  NAME                                        READY   STATUS    RESTARTS   AGE
  #  ingress-nginx-controller-???????????-????   1/1     Running   0          5m8s
  #  lightweb-???????????-????                   1/1     Running   0          4m35s
  #  lightweb-???????????-????                   1/1     Running   0          4m35s
  #  lightweb-???????????-????                   1/1     Running   0          4m35s
  
  kubectl port-forward lightweb-???????????-???? 8080:80
  #  Forwarding from 127.0.0.1:8080 -&gt; 80
  #  Forwarding from [::1]:8080 -&gt; 80
  
  Check in your browser http://localhost:8080
  
  Ctrl+C
  
  </code></pre>
  <h2>Create the ingress object</h2>
  <p>Prepare config.
  Dont forget to point dns record of $YOURDOMAIN to ip shown on end of this tutorial.
  Or simply edit your local hosts file for fake domain:</p>
  <pre><code>cat &lt;&lt; EOF &gt; dummy-ingress.yaml
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: dummy-ingress
    annotations:
      kubernetes.io/ingress.class: &quot;nginx&quot;
  spec:
    rules:
    - host: &quot;$YOURDOMAIN&quot;
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: dummy-service
              port:
                number: 80
  EOF
  </code></pre>
  <p>And apply it:</p>
  <pre><code>kubectl apply -f dummy-ingress.yaml 
  </code></pre>
  <h2>Find the network tags and zone of ingress</h2>
  <pre><code>NETWORK_TAGS=$(gcloud compute instances list --filter=&quot;name=( $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].spec.nodeName}') )&quot; --format=&quot;value(tags.items[0])&quot;) ; echo $NETWORK_TAGS 
  
  NODEZONE=$(gcloud compute instances list --filter=&quot;name=( $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].spec.nodeName}') )&quot; --format=&quot;value(zone)&quot;); echo $NODEZONE 
  </code></pre>
  <h2>Configure the firewall</h2>
  <pre><code>gcloud compute firewall-rules create $CLUSTER_NAME-lb-fw --allow tcp:80 --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags $NETWORK_TAGS 
  </code></pre>
  <h2>Add health check configuration</h2>
  <pre><code>gcloud compute health-checks create http app-service-80-health-check --request-path /healthz --port 80 --check-interval 60 --unhealthy-threshold 3 --healthy-threshold 1 --timeout 5 
  </code></pre>
  <h2>Add the backend service</h2>
  <pre><code>gcloud compute backend-services create $CLUSTER_NAME-lb-backend --health-checks app-service-80-health-check --port-name http --global --enable-cdn --connection-draining-timeout 300 
  </code></pre>
  <h2>Attach our NEG to the backend service</h2>
  <pre><code>gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=$NODEZONE --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global 
  </code></pre>
  <h2>Setup the frontend</h2>
  <pre><code>gcloud compute url-maps create $CLUSTER_NAME-url-map --default-service $CLUSTER_NAME-lb-backend 
  gcloud compute target-http-proxies create $CLUSTER_NAME-http-proxy --url-map $CLUSTER_NAME-url-map 
  gcloud compute forwarding-rules create $CLUSTER_NAME-forwarding-rule --global --ports 80 --target-http-proxy $CLUSTER_NAME-http-proxy 
  </code></pre>
  <h2>enable logging</h2>
  <pre><code>gcloud compute backend-services update $CLUSTER_NAME-lb-backend --enable-logging --global 
  </code></pre>
  <h2>Test</h2>
  <p>Give it some time to deploy ...</p>
  <pre><code>IP_ADDRESS=$(gcloud compute forwarding-rules describe $CLUSTER_NAME-forwarding-rule --global --format=&quot;value(IPAddress)&quot;) ; echo $IP_ADDRESS
  curl -s -I http://$IP_ADDRESS/ #404
  echo curl -s -I http://$YOURDOMAIN/ #200
  </code></pre>
  <h1>cleanup</h1>
  <pre><code># delete the forwarding-rule aka frontend
  gcloud -q compute forwarding-rules delete $CLUSTER_NAME-forwarding-rule --global 
  # delete the http proxy
  gcloud -q compute target-http-proxies delete $CLUSTER_NAME-http-proxy 
  # delete the url map
  gcloud -q compute url-maps delete $CLUSTER_NAME-url-map 
  # delete the backend
  gcloud -q compute backend-services delete $CLUSTER_NAME-lb-backend --global
  # delete the health check
  gcloud -q compute health-checks delete app-service-80-health-check
  # delete the firewall rule
  gcloud -q compute firewall-rules delete $CLUSTER_NAME-lb-fw 
  
  kubectl delete -f dummy-ingress.yaml 
  kubectl delete -f dummy-app-lightweb.yaml 
  helm delete ingress-nginx 
  
  # delete the cluster
  gcloud -q container clusters delete $CLUSTER_NAME --zone=$ZONE
  # delete the NEG  
  gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-a
  gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-b
  gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-c
  gcloud -q compute network-endpoint-groups list
  </code></pre>