title: Ambassador external authentication set-up returns 404
body: <p>I am using Ambassador as the API gateway and am trying to get the external auth to work. I have deployed the auth-service with the corresponding Ambassador annotations</p>
  
  <pre><code>...
  metadata:
    name: auth-service-svc
    annotations:
      getambassador.io/config: |
        ---
        apiVersion: ambassador/v1
        kind:  AuthService
        name:  authentication
        proto: http
        path_prefix: "/api"
        auth_service:  auth-service-svc:8080    
  spec:
    ports:
      - port: 8080
        targetPort: 8080
        name: http
    selector:
      app: auth-service
    type: ClusterIP
  
  </code></pre>
  
  <p>When I do a <code>curl</code> to initiate a Get request against <a href="http://[hostname]/api" rel="nofollow noreferrer">http://[hostname]/api</a>, I expect the auth-service to be called. However this doesn't seem to happen. When I look at the ambassador logs I see the following message:</p>
  
  <pre><code>[2019-04-12 15:10:42.977][75][debug][main] [source/server/connection_handler_impl.cc:257] [C20117] new connection
  [2019-04-12 15:10:42.977][75][debug][http] [source/common/http/conn_manager_impl.cc:243] [C20117] new stream
  [2019-04-12 15:10:42.977][75][debug][http] [source/common/http/conn_manager_impl.cc:580] [C20117][S12856858676043764009] request headers complete (end_stream=true):
  ':authority', '&lt;some-prefix&gt;.elb.amazonaws.com'
  ':path', '/api/'
  ':method', 'GET'
  'user-agent', 'curl/7.61.0'
  'accept', '*/*'
  
  [2019-04-12 15:10:42.977][75][debug][http] [source/common/http/conn_manager_impl.cc:1037] [C20117][S12856858676043764009] request end stream
  [2019-04-12 15:10:42.977][75][debug][router] [source/common/router/router.cc:277] [C20117][S12856858676043764009] no cluster match for URL '/api/'
  [2019-04-12 15:10:42.977][75][debug][http] [source/common/http/conn_manager_impl.cc:1278] [C20117][S12856858676043764009] encoding headers via codec (end_stream=true):
  ':status', '404'
  'date', 'Fri, 12 Apr 2019 15:10:42 GMT'
  'server', 'envoy'
  
  [2019-04-12 15:10:43.013][75][debug][connection] [source/common/network/connection_impl.cc:502] [C20117] remote close
  [2019-04-12 15:10:43.013][75][debug][connection] [source/common/network/connection_impl.cc:183] [C20117] closing socket: 0
  </code></pre>
  
  <p>Here are the logs (truncated) when the auth-service is deleted and re-created. I have tried to show the logs that capture the update of the envoy config.</p>
  
  <pre><code>...
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: copying configuration from http://localhost:34525/api/snapshot/40 to /ambassador/snapshots/snapshot-tmp.yaml
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Load balancer for tcp://auth-service-svc:8080 is None
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Load balancer for tcp://127.0.0.1:8877 is None
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Load balancer for tcp://127.0.0.1:8877 is None
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Load balancer for tcp://127.0.0.1:8877 is None
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: extauth: server_uri http://api
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: listen_ports ['80']
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: -global-: NOTICE: Ambassador 0.60 will default to listening on port 8080 for HTTP. You will need to change your configuration to continue using port 80.
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: successfully validated the resulting envoy configuration, continuing...
  ....
  
  [2019-04-12 15:20:59.782][65][debug][config] [source/server/listener_manager_impl.cc:56]     name: envoy.http_connection_manager
  [2019-04-12 15:20:59.783][65][debug][config] [source/server/listener_manager_impl.cc:59]   config: {"use_remote_address":true,"access_log":[{"config":{"path":"/dev/fd/1","format":"ACCESS [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n"},"name":"envoy.file_access_log"}],"xff_num_trusted_hops":0,"normalize_path":true,"route_config":{"virtual_hosts":[{"name":"backend","routes":[{"route":{"priority":null,"prefix_rewrite":"/ambassador/v0/check_ready","timeout":"10.000s","weighted_clusters":{"clusters":[{"name":"cluster_127_0_0_1_8877","weight":100}]}},"match":{"prefix":"/ambassador/v0/check_ready","case_sensitive":true}},{"match":{"case_sensitive":true,"prefix":"/ambassador/v0/check_alive"},"route":{"priority":null,"prefix_rewrite":"/ambassador/v0/check_alive","timeout":"10.000s","weighted_clusters":{"clusters":[{"name":"cluster_127_0_0_1_8877","weight":100}]}}},{"match":{"case_sensitive":true,"prefix":"/ambassador/v0/"},"route":{"priority":null,"weighted_clusters":{"clusters":[{"weight":100,"name":"cluster_127_0_0_1_8877"}]},"timeout":"10.000s","prefix_rewrite":"/ambassador/v0/"}},{"match":{"case_sensitive":true,"prefix":"/leads/"},"route":{"timeout":"3.000s","prefix_rewrite":"/","weighted_clusters":{"clusters":[{"weight":100,"name":"cluster_lead_service_svc"}]},"priority":null}}],"domains":["*"]}]},"http_filters":[{"config":{"http_service":{"server_uri":{"timeout":"5.000s","uri":"http://api","cluster":"cluster_extauth_auth_service_svc_8080"},"authorization_request":{"allowed_headers":{"patterns":[{"exact":"x-forwarded-proto"},{"exact":"cookie"},{"exact":"user-agent"},{"exact":"proxy-authorization"},{"exact":"from"},{"exact":"authorization"},{"exact":"x-forwarded-for"},{"exact":"x-forwarded-host"}]}},"path_prefix":"/api","authorization_response":{"allowed_client_headers":{"patterns":[{"exact":"authorization"},{"exact":"set-cookie"},{"exact":"location"},{"exact":"www-authenticate"},{"exact":"proxy-authenticate"}]},"allowed_upstream_headers":{"patterns":[{"exact":"authorization"},{"exact":"set-cookie"},{"exact":"location"},{"exact":"www-authenticate"},{"exact":"proxy-authenticate"}]}}}},"name":"envoy.ext_authz"},{"name":"envoy.cors"},{"name":"envoy.router"}],"stat_prefix":"ingress_http"}
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: configuration updated from snapshot 40
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Scout reports {"latest_version": "0.53.1", "application": "ambassador", "cached": true, "timestamp": 1555081685.663466}
  2019-04-12 15:20:59 diagd 0.53.1 [P43TAmbassadorEventWatcher] INFO: Scout notices: [{"level": "DEBUG", "message": "Returning cached result"}]
  [2019-04-12 15:20:59.799][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:302]     http filter #0
  [2019-04-12 15:20:59.799][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:303]       name: envoy.ext_authz
  [2019-04-12 15:20:59.800][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:307]     config: {"http_service":{"authorization_response":{"allowed_upstream_headers":{"patterns":[{"exact":"authorization"},{"exact":"set-cookie"},{"exact":"location"},{"exact":"www-authenticate"},{"exact":"proxy-authenticate"}]},"allowed_client_headers":{"patterns":[{"exact":"authorization"},{"exact":"set-cookie"},{"exact":"location"},{"exact":"www-authenticate"},{"exact":"proxy-authenticate"}]}},"server_uri":{"timeout":"5.000s","uri":"http://api","cluster":"cluster_extauth_auth_service_svc_8080"},"authorization_request":{"allowed_headers":{"patterns":[{"exact":"x-forwarded-proto"},{"exact":"cookie"},{"exact":"user-agent"},{"exact":"proxy-authorization"},{"exact":"from"},{"exact":"authorization"},{"exact":"x-forwarded-for"},{"exact":"x-forwarded-host"}]}},"path_prefix":"/api"}}
  [2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:302]     http filter #1
  [2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:303]       name: envoy.cors
  [2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:307]     config: {}
  [2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:302]     http filter #2
  [2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:303]       name: envoy.router
  [2019-04-12 15:20:59.801][65][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:307]     config: {}
  [2019-04-12 15:20:59.804][65][debug][config] [source/server/listener_manager_impl.cc:627] add warming listener: name=ambassador-listener-80, hash=1783155174245818883, address=0.0.0.0:80
  [2019-04-12 15:20:59.804][65][debug][init] [source/common/init/manager_impl.cc:45] init manager Listener ambassador-listener-80 contains no targets
  [2019-04-12 15:20:59.804][65][debug][init] [source/common/init/watcher_impl.cc:14] init manager Listener ambassador-listener-80 initialized, notifying ListenerImpl
  [2019-04-12 15:20:59.805][65][debug][config] [source/server/listener_manager_impl.cc:627] warm complete. updating active listener: name=ambassador-listener-80, hash=1783155174245818883, address=0.0.0.0:80
  [2019-04-12 15:20:59.805][65][debug][config] [source/server/listener_manager_impl.cc:627] draining listener: name=ambassador-listener-80, hash=5292162044335998753, address=0.0.0.0:80
  [2019-04-12 15:20:59.805][65][info][upstream] [source/server/lds_api.cc:74] lds: add/update listener 'ambassador-listener-80'
  [2019-04-12 15:20:59.806][65][debug][config] [source/common/config/grpc_mux_impl.cc:104] Resuming discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration
  [2019-04-12 15:20:59.806][65][debug][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC config for type.googleapis.com/envoy.api.v2.Listener accepted with 1 resources with version v40
  [2019-04-12 15:20:59.806][65][debug][config] [source/common/config/grpc_mux_impl.cc:118] Received gRPC message for type.googleapis.com/envoy.api.v2.Cluster at version v40
  [2019-04-12 15:20:59.806][65][debug][config] [source/common/config/grpc_mux_impl.cc:96] Pausing discovery requests for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
  [2019-04-12 15:20:59.811][65][info][upstream] [source/common/upstream/cluster_manager_impl.cc:483] add/update cluster cluster_extauth_auth_service_svc_8080 starting warming
  [2019-04-12 15:20:59.811][65][debug][config] [source/common/config/grpc_mux_impl.cc:96] Pausing discovery requests for type.googleapis.com/envoy.api.v2.Cluster
  [2019-04-12 15:20:59.811][65][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
  [2019-04-12 15:20:59.811][65][debug][upstream] [source/common/upstream/cds_api_impl.cc:110] cds: add/update cluster 'cluster_extauth_auth_service_svc_8080'
  [2019-04-12 15:20:59.811][65][debug][config] [source/common/config/grpc_mux_impl.cc:104] Resuming discovery requests for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
  [2019-04-12 15:20:59.811][65][debug][config] [bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC config for type.googleapis.com/envoy.api.v2.Cluster accepted with 3 resources with version v40
  [2019-04-12 15:20:59.812][65][debug][upstream] [source/common/upstream/upstream_impl.cc:1358] DNS hosts have changed for auth-service-svc
  [2019-04-12 15:20:59.812][65][debug][upstream] [source/common/upstream/upstream_impl.cc:721] initializing secondary cluster cluster_extauth_auth_service_svc_8080 completed
  [2019-04-12 15:20:59.812][65][debug][init] [source/common/init/manager_impl.cc:45] init manager Cluster cluster_extauth_auth_service_svc_8080 contains no targets
  [2019-04-12 15:20:59.812][65][debug][init] [source/common/init/watcher_impl.cc:14] init manager Cluster cluster_extauth_auth_service_svc_8080 initialized, notifying ClusterImplBase
  [2019-04-12 15:20:59.812][65][info][upstream] [source/common/upstream/cluster_manager_impl.cc:496] warming cluster cluster_extauth_auth_service_svc_8080 complete
  [2019-04-12 15:20:59.812][75][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:517] adding TLS cluster cluster_extauth_auth_service_svc_8080
  [2019-04-12 15:20:59.812][76][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:517] adding TLS cluster cluster_extauth_auth_service_svc_8080
  [2019-04-12 15:20:59.812][65][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:517] adding TLS cluster cluster_extauth_auth_service_svc_8080
  [2019-04-12 15:20:59.812][75][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:978] membership update for TLS cluster cluster_extauth_auth_service_svc_8080 added 1 removed 0
  [2019-04-12 15:20:59.812][76][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:978] membership update for TLS cluster cluster_extauth_auth_service_svc_8080 added 1 removed 0
  [2019-04-12 15:20:59.812][65][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:978] membership update for TLS cluster cluster_extauth_auth_service_svc_8080 added 1 removed 0
  [2019-04-12 15:20:59.812][65][debug][config] [source/common/config/grpc_mux_impl.cc:104] Resuming discovery requests for type.googleapis.com/envoy.api.v2.Cluster
  ....
  
  </code></pre>
  
  <p>Any hints as to why the path is not being detected and routed to the external auth service?</p>