Issue title: Terminated rpc error: code = Canceled desc = context canceled
 Issue body: ### Bug Description

Command 'istioctl ps' can't find 'deploy-d' service that injected sidecar. It is istiod log:
`
2022-07-13T16:29:53.362717Z	info	ads	CDS: PUSH for node:deploy-d-784c47df75-l7fth.asm-e2e-demo-ns resources:58 size:54.9kB cached:52/52
2022-07-13T16:29:53.366905Z	info	ads	EDS: PUSH for node:deploy-d-784c47df75-l7fth.asm-e2e-demo-ns resources:46 size:13.9kB empty:0 cached:46/46
2022-07-13T16:58:57.807818Z	info	ads	ADS: "116.69.203.115:53194" deploy-d-784c47df75-l7fth.asm-e2e-demo-ns-11 terminated rpc error: code = Canceled desc = context canceled
2022-07-13T16:58:57.811934Z	info	ads	Client closed connection deploy-d-784c47df75-l7fth.asm-e2e-demo-ns-11
`
Finally it is work after restarting the pod.


### Version

```prose
client version: 1.12.4
control plane version: 1.12.6
data plane version: 1.12.6-v3.10-250-cb5a38dedfec569e07300 (16 proxies)
```


### Additional Information

_No response_

### Affected product area

- [ ] Docs
- [ ] Installation
- [X] Networking
- [X] Performance and Scalability
- [ ] Extensions and Telemetry
- [ ] Security
- [ ] Test and Release
- [ ] User Experience
- [ ] Developer Infrastructure
- [ ] Upgrade
- [ ] Multi Cluster
- [ ] Virtual Machine
- [ ] Control Plane Revisions

### Is this the right place to submit this?

- [X] This is not a security vulnerability
- [X] This is not a question about how to use Istio
 Comments: 
 Comment 0: Update:

We encounter this issue occasionally when the network jitters (for example eth down / up). Then:

1. istioctl ps cannot list the pod
2. When exec into the pod, using tools like `ss` or `netstat`, there is no connection to the istiod (from pilot-agent).

So it seems the xds proxy did not reconnect occasionally.

One thing that fun:

When the xds proxy is stuck, exec into the istio-proxy container and execute:

```
# Send something junk to the debug port
curl -XPOST http://localhost:15004/debug/logging
```

Then the xds proxy will start again:

```
022-08-20T13:57:23.875076Z	error	xdsproxy	upstream [1571] send error for type url istio.io/debug/logging: EOF
2022-08-20T13:57:23.876239Z	error	xdsproxy	downstream [1571] send error: rpc error: code = Canceled desc = context canceled
2022-08-20T13:57:23.876581Z	error	xdsproxy	downstream [1571] send error: rpc error: code = Canceled desc = context canceled
2022-08-20T13:57:23.877016Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 2, EOF
2022-08-20T13:57:24.268793Z	info	xdsproxy	connected to upstream XDS server: istiod-1-12.istio-system.svc:15012
```
 Comment 1: I believe there is something wrong in the bi-directional grpc. All symptoms indicate that maybe we are stuck waiting during `con.upstream.Recv()`.

Any thoughts? @hzxuzhonghu @howardjohn


 Comment 2: Here is the dump of `/debug/pprof/goroutine?debug=2` from another pod which disconnected from istiod and blocks on Recv (not reconnect again):

<details>

```
goroutine 285090 [running]:
runtime/pprof.writeGoroutineStacks(0x21405a0, 0xc0007107e0, 0x30, 0x1c635e0)
	runtime/pprof/pprof.go:693 +0x9f
runtime/pprof.writeGoroutine(0x21405a0, 0xc0007107e0, 0x2, 0xc001280800, 0x0)
	runtime/pprof/pprof.go:682 +0x45
runtime/pprof.(*Profile).WriteTo(0x2f301e0, 0x21405a0, 0xc0007107e0, 0x2, 0xc0007107e0, 0xc000a6d8b8)
	runtime/pprof/pprof.go:331 +0x3f2
net/http/pprof.handler.ServeHTTP(0xc000940401, 0x9, 0x2172420, 0xc0007107e0, 0xc0000e7c00)
	net/http/pprof/pprof.go:253 +0x385
net/http/pprof.Index(0x2172420, 0xc0007107e0, 0xc0000e7c00)
	net/http/pprof/pprof.go:371 +0x8d7
istio.io/istio/pilot/cmd/pilot-agent/status.(*Server).handlePprofIndex(0xc00060a300, 0x2172420, 0xc0007107e0, 0xc0000e7c00)
	istio.io/istio/pilot/cmd/pilot-agent/status/server.go:358 +0x58
net/http.HandlerFunc.ServeHTTP(0xc0008133f0, 0x2172420, 0xc0007107e0, 0xc0000e7c00)
	net/http/server.go:2049 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0007bc700, 0x2172420, 0xc0007107e0, 0xc0000e7c00)
	net/http/server.go:2428 +0x1ad
net/http.serverHandler.ServeHTTP(0xc00095c000, 0x2172420, 0xc0007107e0, 0xc0000e7c00)
	net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc0008b8280, 0x2175a28, 0xc000b80540)
	net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
	net/http/server.go:2993 +0x39b

goroutine 1 [semacquire, 8578 minutes]:
sync.runtime_Semacquire(0xc0006280e0)
	runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0006280d8)
	sync/waitgroup.go:130 +0x65
main.glob..func1(0x2f41720, 0xc0008ac780, 0x1, 0x8, 0x0, 0x0)
	istio.io/istio/pilot/cmd/pilot-agent/main.go:160 +0x7d2
github.com/spf13/cobra.(*Command).execute(0x2f41720, 0xc0008ac700, 0x8, 0x8, 0x2f41720, 0xc0008ac700)
	mcclainmichael@example.org/command.go:856 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0x2f414a0, 0x41af01, 0x0, 0x0)
	mcclainmichael@example.org/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	mcclainmichael@example.org/command.go:902
main.main()
	istio.io/istio/pilot/cmd/pilot-agent/main.go:306 +0x36

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x2f5b6c0)
	mcclainmichael@example.org/klog.go:1169 +0x8b
created by k8s.io/klog/v2.init.0
	mcclainmichael@example.org/klog.go:420 +0xdf

goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000066600)
	mcclainmichael@example.org/stats/view/worker.go:276 +0xcd
created by go.opencensus.io/stats/view.init.0
	mcclainmichael@example.org/stats/view/worker.go:34 +0x68

goroutine 86 [syscall, 8578 minutes]:
os/signal.signal_recv(0xc00088aeb8)
	runtime/sigqueue.go:168 +0xa5
os/signal.loop()
	os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
	os/signal/signal.go:151 +0x45

goroutine 58 [IO wait, 8578 minutes]:
internal/poll.runtime_pollWait(0x7f06e2f5ccf0, 0x72, 0x0)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc00060a598, 0x72, 0x0, 0x0, 0x1e356ef)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00060a580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc00060a580, 0xc, 0x7f0709c035b8, 0x10)
	net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000455308, 0x40e2f8, 0xc, 0x1c047a0)
	net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000455308, 0xc00055b501, 0xc0000946c8, 0x47d6fa, 0xc00055b5d4)
	net/tcpsock.go:261 +0x65
github.com/miekg/dns.(*Server).serveTCP(0xc0007c0a20, 0x2172210, 0xc000455308, 0x0, 0x0)
	mcclainmichael@example.org/server.go:445 +0x118
github.com/miekg/dns.(*Server).ActivateAndServe(0xc0007c0a20, 0x0, 0x0)
	mcclainmichael@example.org/server.go:363 +0x1bd
istio.io/istio/pkg/dns/client.(*dnsProxy).start(0xc000614630)
	istio.io/istio/pkg/dns/client/proxy.go:66 +0x33
created by istio.io/istio/pkg/dns/client.(*LocalDNSServer).StartDNS
	istio.io/istio/pkg/dns/client/dns.go:179 +0x5e

goroutine 54 [chan receive, 8578 minutes]:
istio.io/istio/pilot/cmd/pilot-agent/status.(*Server).Run(0xc00060a300, 0x2175980, 0xc000606d00)
	istio.io/istio/pilot/cmd/pilot-agent/status/server.go:348 +0x70e
created by main.initStatusServer
	istio.io/istio/pilot/cmd/pilot-agent/main.go:218 +0x172

goroutine 84 [IO wait]:
internal/poll.runtime_pollWait(0x7f06e2f5cb20, 0x72, 0x0)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0008ad418, 0x72, 0x0, 0x0, 0x1e356ef)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0008ad400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc0008ad400, 0xdfd2a57db5133b89, 0x0, 0x0)
	net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc00088aeb8, 0x6301cf2a, 0xc0000c7e08, 0x497da6)
	net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc00088aeb8, 0xc0000c7e58, 0x18, 0xc0008a2780, 0xa0d7bb)
	net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc00095c000, 0x2172210, 0xc00088aeb8, 0x0, 0x0)
	net/http/server.go:2961 +0x285
net/http.Serve(...)
	net/http/server.go:2509
istio.io/istio/pilot/cmd/pilot-agent/status.(*Server).Run.func1(0x2172210, 0xc00088aeb8, 0xc0007bc700, 0x2175980, 0xc000606d00)
	istio.io/istio/pilot/cmd/pilot-agent/status/server.go:333 +0x74
created by istio.io/istio/pilot/cmd/pilot-agent/status.(*Server).Run
	istio.io/istio/pilot/cmd/pilot-agent/status/server.go:332 +0x6dd

goroutine 56 [chan receive, 8578 minutes]:
istio.io/istio/pkg/cmd.WaitSignalFunc(0xc0000acd00)
	istio.io/istio/pkg/cmd/cmd.go:42 +0xb6
created by main.glob..func1
	istio.io/istio/pilot/cmd/pilot-agent/main.go:153 +0x725

goroutine 57 [IO wait]:
internal/poll.runtime_pollWait(0x7f06e2f5cc08, 0x72, 0x28)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc00060a418, 0x72, 0x200, 0x200, 0xc000d69d70)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadMsg(0xc00060a400, 0xc000b24600, 0x200, 0x200, 0xc000d69d70, 0x28, 0x28, 0x0, 0x0, 0x0, ...)
	internal/poll/fd_unix.go:249 +0x252
net.(*netFD).readMsg(0xc00060a400, 0xc000b24600, 0x200, 0x200, 0xc000d69d70, 0x28, 0x28, 0xc0000c8b48, 0x441272, 0xc0000c8b88, ...)
	net/fd_posix.go:67 +0x90
net.(*UDPConn).readMsg(0xc00068e630, 0xc000b24600, 0x200, 0x200, 0xc000d69d70, 0x28, 0x28, 0x7f0709c035b8, 0x30, 0xc000d69d70, ...)
	net/udpsock_posix.go:59 +0x91
net.(*UDPConn).ReadMsgUDP(0xc00068e630, 0xc000b24600, 0x200, 0x200, 0xc000d69d70, 0x28, 0x28, 0x47dbc5, 0x1, 0xc0000c8d10, ...)
	net/udpsock.go:139 +0x9d
github.com/miekg/dns.ReadFromSessionUDP(0xc00068e630, 0xc000b24600, 0x200, 0x200, 0x0, 0x0, 0x2f5b260, 0xc000a7b400)
	mcclainmichael@example.org/udp.go:42 +0xb0
github.com/miekg/dns.(*Server).readUDP(0xc0007c07e0, 0xc00068e630, 0x77359400, 0x2f8a9e0, 0x213f901, 0x2f8a9e0, 0x7f06e2f90670, 0x2f8a9e0, 0x7f06e2f90601)
	mcclainmichael@example.org/server.go:687 +0xd9
github.com/miekg/dns.defaultReader.ReadUDP(0xc0007c07e0, 0xc00068e630, 0x77359400, 0x7f06e2f90640, 0xc000a7b400, 0x1, 0x0, 0x21402a0, 0xc000a7b400)
	mcclainmichael@example.org/server.go:174 +0x45
github.com/miekg/dns.(*Server).serveUDP(0xc0007c07e0, 0x2183718, 0xc00068e630, 0x0, 0x0)
	mcclainmichael@example.org/server.go:501 +0x217
github.com/miekg/dns.(*Server).ActivateAndServe(0xc0007c07e0, 0x0, 0x0)
	mcclainmichael@example.org/server.go:358 +0x105
istio.io/istio/pkg/dns/client.(*dnsProxy).start(0xc0006144e0)
	istio.io/istio/pkg/dns/client/proxy.go:66 +0x33
created by istio.io/istio/pkg/dns/client.(*LocalDNSServer).StartDNS
	istio.io/istio/pkg/dns/client/dns.go:179 +0x5e

goroutine 59 [select, 627 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000517540)
	mcclainmichael@example.org/balancer_conn_wrappers.go:77 +0xb2
created by google.golang.org/grpc.newCCBalancerWrapper
	mcclainmichael@example.org/balancer_conn_wrappers.go:67 +0x1e6

goroutine 61 [syscall, 8578 minutes]:
syscall.Syscall6(0xe8, 0xa, 0xc000a0fc2c, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
	syscall/asm_linux_amd64.s:43 +0x5
golang.org/x/sys/unix.EpollWait(0xa, 0xc000a0fc2c, 0x7, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	mcclainmichael@example.org/unix/zsyscall_linux_amd64.go:56 +0x72
github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc0004944c0, 0x0, 0x0, 0x0)
	mcclainmichael@example.org/inotify_poller.go:87 +0x91
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000517630)
	mcclainmichael@example.org/inotify.go:193 +0x206
created by github.com/fsnotify/fsnotify.NewWatcher
	mcclainmichael@example.org/inotify.go:60 +0x1ab

goroutine 62 [select, 657 minutes]:
istio.io/istio/pkg/queue.(*delayQueue).Run(0xc000614ba0, 0xc00069cba0)
	istio.io/istio/pkg/queue/delay.go:200 +0x235
created by istio.io/istio/security/pkg/nodeagent/cache.NewSecretManagerClient
	istio.io/istio/security/pkg/nodeagent/cache/secretcache.go:189 +0x235

goroutine 63 [select, 8578 minutes]:
istio.io/istio/security/pkg/nodeagent/cache.(*SecretManagerClient).handleFileWatch(0xc0007c0ea0)
	istio.io/istio/security/pkg/nodeagent/cache/secretcache.go:651 +0xdb
created by istio.io/istio/security/pkg/nodeagent/cache.NewSecretManagerClient
	istio.io/istio/security/pkg/nodeagent/cache/secretcache.go:190 +0x257

goroutine 177 [select, 57 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0008a3b00)
	mcclainmichael@example.org/internal/transport/http2_server.go:1119 +0x265
created by google.golang.org/grpc/internal/transport.NewServerTransport
	mcclainmichael@example.org/internal/transport/http2_server.go:331 +0x1295

goroutine 46 [select, 657 minutes]:
istio.io/istio/pkg/queue.(*delayQueue).work(0xc000614ba0, 0xc00069cba0)
	istio.io/istio/pkg/queue/delay.go:232 +0xa5
created by istio.io/istio/pkg/queue.(*delayQueue).Run
	istio.io/istio/pkg/queue/delay.go:180 +0x65

goroutine 87 [select, 19 minutes]:
istio.io/istio/pilot/pkg/model.(*JwksResolver).refresher(0xc00060a780)
	istio.io/istio/pilot/pkg/model/jwks_resolver.go:379 +0xcf
created by istio.io/istio/pilot/pkg/model.newJwksResolverWithCABundlePaths
	istio.io/istio/pilot/pkg/model/jwks_resolver.go:217 +0x277

goroutine 88 [select, 8578 minutes]:
istio.io/istio/pilot/pkg/config/memory.(*configStoreMonitor).Run(0xc000deeab0, 0xc00069cc00)
	istio.io/istio/pilot/pkg/config/memory/monitor.go:93 +0xfc
istio.io/istio/pilot/pkg/config/memory.(*Controller).Run(0xc000deeae0, 0xc00069cc00)
	istio.io/istio/pilot/pkg/config/memory/controller.go:80 +0x3d
created by istio.io/istio/pilot/pkg/xds.NewXDS
	istio.io/istio/pilot/pkg/xds/simple.go:119 +0xadc

goroutine 90 [select, 8578 minutes]:
istio.io/istio/pilot/pkg/xds.debounce(0xc0007506c0, 0xc00069cc00, 0x5f5e100, 0x2540be400, 0x2175901, 0xc00055e5c0, 0xc000610890)
	istio.io/istio/pilot/pkg/xds/discovery.go:450 +0x273
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).handleUpdates(0xc000612b00, 0xc00069cc00)
	istio.io/istio/pilot/pkg/xds/discovery.go:399 +0x9b
created by istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).Start
	istio.io/istio/pilot/pkg/xds/discovery.go:269 +0x85

goroutine 91 [select]:
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).periodicRefreshMetrics(0xc000612b00, 0xc00069cc00)
	istio.io/istio/pilot/pkg/xds/discovery.go:301 +0xc9
created by istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).Start
	istio.io/istio/pilot/pkg/xds/discovery.go:270 +0xb1

goroutine 92 [sync.Cond.Wait, 657 minutes]:
sync.runtime_notifyListWait(0xc000607110, 0xc00000000b)
	runtime/sema.go:513 +0xf8
sync.(*Cond).Wait(0xc000607100)
	sync/cond.go:56 +0x99
istio.io/istio/pilot/pkg/xds.(*PushQueue).Dequeue(0xc000607140, 0x0, 0x0, 0xc0000b6200)
	istio.io/istio/pilot/pkg/xds/pushqueue.go:83 +0x90
istio.io/istio/pilot/pkg/xds.doSendPushes(0xc00069cc00, 0xc00069cc60, 0xc000607140)
	istio.io/istio/pilot/pkg/xds/discovery.go:510 +0xac
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).sendPushes(0xc000612b00, 0xc00069cc00)
	istio.io/istio/pilot/pkg/xds/discovery.go:575 +0x45
created by istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).Start
	istio.io/istio/pilot/pkg/xds/discovery.go:271 +0xdd

goroutine 94 [IO wait, 8578 minutes]:
internal/poll.runtime_pollWait(0x7f06e2f5ca38, 0x72, 0x0)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000bad418, 0x72, 0x0, 0x0, 0x1e356ef)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000bad400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc000bad400, 0xc000c84030, 0x0, 0x0)
	net/fd_unix.go:172 +0x45
net.(*UnixListener).accept(0xc000deef60, 0xc0009f4e38, 0xc0009f4e40, 0x28)
	net/unixsock_posix.go:162 +0x32
net.(*UnixListener).Accept(0xc000deef60, 0x1edebe0, 0xc000bbb180, 0x2172240, 0xc000deef60)
	net/unixsock.go:260 +0x65
google.golang.org/grpc.(*Server).Serve(0xc000bbb180, 0x2172240, 0xc000deef60, 0x0, 0x0)
	mcclainmichael@example.org/server.go:779 +0x27f
istio.io/istio/security/pkg/nodeagent/sds.(*Server).initWorkloadSdsService.func1(0xc000614c00, 0xc0007a4340, 0xc000dc5d50)
	istio.io/istio/security/pkg/nodeagent/sds/server.go:111 +0x1c5
created by istio.io/istio/security/pkg/nodeagent/sds.(*Server).initWorkloadSdsService
	istio.io/istio/security/pkg/nodeagent/sds/server.go:94 +0x1de

goroutine 95 [select, 9 minutes]:
istio.io/istio/pkg/wasm.(*LocalFileCache).purge(0xc000dc0840)
	istio.io/istio/pkg/wasm/cache.go:229 +0x3fd
istio.io/istio/pkg/wasm.NewLocalFileCache.func1(0xc000dc0840)
	istio.io/istio/pkg/wasm/cache.go:99 +0x2b
created by istio.io/istio/pkg/wasm.NewLocalFileCache
	istio.io/istio/pkg/wasm/cache.go:98 +0x138

goroutine 96 [IO wait, 5783 minutes]:
internal/poll.runtime_pollWait(0x7f06e2f5c868, 0x72, 0x0)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000bad598, 0x72, 0x0, 0x0, 0x1e356ef)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000bad580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc000bad580, 0xc000c841b0, 0x0, 0x0)
	net/fd_unix.go:172 +0x45
net.(*UnixListener).accept(0xc000def080, 0xc0009f3e78, 0xc0009f3e80, 0x28)
	net/unixsock_posix.go:162 +0x32
net.(*UnixListener).Accept(0xc000def080, 0x1edebe0, 0xc000bbb340, 0x2172240, 0xc000def080)
	net/unixsock.go:260 +0x65
google.golang.org/grpc.(*Server).Serve(0xc000bbb340, 0x2172240, 0xc000def080, 0x0, 0x0)
	mcclainmichael@example.org/server.go:779 +0x27f
istio.io/istio/pkg/istio-agent.initXdsProxy.func3(0xc000aff760)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:195 +0x45
created by istio.io/istio/pkg/istio-agent.initXdsProxy
	istio.io/istio/pkg/istio-agent/xds_proxy.go:194 +0x490

goroutine 131 [IO wait, 8578 minutes]:
internal/poll.runtime_pollWait(0x7f06e2f5c780, 0x72, 0x0)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000bad818, 0x72, 0x0, 0x0, 0x1e356ef)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000bad800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc000bad800, 0x30, 0x30, 0x7f0709c03f18)
	net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000df44e0, 0xc00061dde8, 0x40e2f8, 0x30)
	net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000df44e0, 0x1cdaa40, 0xc000a5e120, 0x1aff520, 0x2f1ffd0)
	net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc000df02a0, 0x2172210, 0xc000df44e0, 0x0, 0x0)
	net/http/server.go:2961 +0x285
istio.io/istio/pkg/istio-agent.(*XdsProxy).initDebugInterface.func1(0x2172210, 0xc000df44e0, 0xc000aff760)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:854 +0xd5
created by istio.io/istio/pkg/istio-agent.(*XdsProxy).initDebugInterface
	istio.io/istio/pkg/istio-agent/xds_proxy.go:852 +0x2ab

goroutine 179 [select, 657 minutes]:
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).Stream(0xc000612b00, 0x7f06e2e7ccb8, 0xc00055f140, 0x7f06e2e7ccb8, 0xc00055f140)
	istio.io/istio/pilot/pkg/xds/ads.go:321 +0x77f
istio.io/istio/security/pkg/nodeagent/sds.(*sdsservice).StreamSecrets(0xc000614c30, 0x21855e0, 0xc00055f140, 0x2171670, 0xc000614c30)
	istio.io/istio/security/pkg/nodeagent/sds/sdsservice.go:189 +0x75
github.com/envoyproxy/go-control-plane/envoy/service/secret/v3._SecretDiscoveryService_StreamSecrets_Handler(0x1c93c00, 0xc000614c30, 0x217e320, 0xc00047c900, 0x1a199a0, 0xc00055f130)
	mcclainmichael@example.org/envoy/service/secret/v3/sds.pb.go:355 +0xad
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000bbb180, 0x21890f8, 0xc0008a3b00, 0xc0007c0000, 0xc000deef00, 0x2f34540, 0x0, 0x0, 0x0)
	mcclainmichael@example.org/server.go:1540 +0xee5
google.golang.org/grpc.(*Server).handleStream(0xc000bbb180, 0x21890f8, 0xc0008a3b00, 0xc0007c0000, 0x0)
	mcclainmichael@example.org/server.go:1620 +0xca5
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00055a040, 0xc000bbb180, 0x21890f8, 0xc0008a3b00, 0xc0007c0000)
	mcclainmichael@example.org/server.go:921 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	mcclainmichael@example.org/server.go:919 +0x1fd

goroutine 133 [select, 8578 minutes]:
istio.io/istio/pkg/envoy.(*Agent).Run(0xc0000f0e70, 0x2175980, 0xc000606d00)
	istio.io/istio/pkg/envoy/agent.go:113 +0x172
istio.io/istio/pkg/istio-agent.(*Agent).Run.func1(0xc000628000, 0x2175980, 0xc000606d00)
	istio.io/istio/pkg/istio-agent/agent.go:463 +0xa5
created by istio.io/istio/pkg/istio-agent.(*Agent).Run
	istio.io/istio/pkg/istio-agent/agent.go:443 +0x576

goroutine 134 [select, 8578 minutes]:
istio.io/istio/pkg/envoy.(*envoy).Run(0xc000e29080, 0x0, 0xc000e8cc00, 0x2, 0x0)
	istio.io/istio/pkg/envoy/proxy.go:192 +0x2f3
istio.io/istio/pkg/envoy.(*Agent).runWait(0xc0000f0e70, 0x0, 0xc000e8cc00)
	istio.io/istio/pkg/envoy/agent.go:208 +0xcc
created by istio.io/istio/pkg/envoy.(*Agent).Run
	istio.io/istio/pkg/envoy/agent.go:111 +0xc5

goroutine 135 [syscall, 8578 minutes]:
syscall.Syscall6(0xf7, 0x1, 0x11, 0xc000492600, 0x1000004, 0x0, 0x0, 0xc000492618, 0x472427, 0x47d7fc)
	syscall/asm_linux_amd64.s:43 +0x5
os.(*Process).blockUntilWaitable(0xc000e822d0, 0xc00022eab0, 0x4, 0xc00022eab0)
	os/wait_waitid.go:32 +0x9e
os.(*Process).wait(0xc000e822d0, 0xc000492798, 0x2, 0x2)
	os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
	os/exec.go:129
os/exec.(*Cmd).Wait(0xc000affce0, 0x2111b20, 0x1a84280)
	os/exec/exec.go:507 +0x65
istio.io/istio/pkg/envoy.(*envoy).Run.func1(0xc000e8cde0, 0xc000affce0)
	istio.io/istio/pkg/envoy/proxy.go:189 +0x2b
created by istio.io/istio/pkg/envoy.(*envoy).Run
	istio.io/istio/pkg/envoy/proxy.go:188 +0x265

goroutine 111307 [IO wait]:
internal/poll.runtime_pollWait(0x7f06e2f5cdd8, 0x72, 0xffffffffffffffff)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000ab6618, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000ab6600, 0xc000158000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000ab6600, 0xc000158000, 0x8000, 0x8000, 0xad7b7e, 0x800010601, 0x0)
	net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000c841b0, 0xc000158000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	net/net.go:183 +0x91
bufio.(*Reader).Read(0xc000c80540, 0xc00095c4a0, 0x9, 0x9, 0x14, 0x7f0709c043c8, 0x18)
	bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x2132bc0, 0xc000c80540, 0xc00095c4a0, 0x9, 0x9, 0x9, 0xacfcc5, 0xc000a364d4, 0xc0004d6000)
	io/io.go:328 +0x87
io.ReadFull(...)
	io/io.go:347
golang.org/x/net/http2.readFrameHeader(0xc00095c4a0, 0x9, 0x9, 0x2132bc0, 0xc000c80540, 0x0, 0x0, 0x215b160, 0xc000a364c8)
	mcclainmichael@example.org/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00095c460, 0xc000a364c8, 0x2f5b260, 0x0, 0x0)
	mcclainmichael@example.org/http2/frame.go:498 +0xa5
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0009e6480, 0xc000e4e2a0, 0x1edec20)
	mcclainmichael@example.org/internal/transport/http2_server.go:599 +0x9b
google.golang.org/grpc.(*Server).serveStreams(0xc000bbb340, 0x21890f8, 0xc0009e6480)
	mcclainmichael@example.org/server.go:905 +0x103
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000bbb340, 0x21890f8, 0xc0009e6480, 0xc000d9ecf0, 0x13)
	mcclainmichael@example.org/server.go:847 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	mcclainmichael@example.org/server.go:846 +0x190

goroutine 64 [select, 57 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0005da2d0, 0x1, 0x0, 0x0, 0x0, 0x0)
	mcclainmichael@example.org/internal/transport/controlbuf.go:407 +0xff
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000c81740, 0x0, 0x0)
	mcclainmichael@example.org/internal/transport/controlbuf.go:534 +0x1dd
google.golang.org/grpc/internal/transport.NewServerTransport.func2(0xc0008a3b00)
	mcclainmichael@example.org/internal/transport/http2_server.go:322 +0xd7
created by google.golang.org/grpc/internal/transport.NewServerTransport
	mcclainmichael@example.org/internal/transport/http2_server.go:319 +0x1270

goroutine 178 [IO wait, 57 minutes]:
internal/poll.runtime_pollWait(0x7f06e2f5c698, 0x72, 0xffffffffffffffff)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000ab6418, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000ab6400, 0xc000c42000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000ab6400, 0xc000c42000, 0x8000, 0x8000, 0xad7b7e, 0x800010601, 0x0)
	net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000c84030, 0xc000c42000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	net/net.go:183 +0x91
bufio.(*Reader).Read(0xc000ba2ba0, 0xc000710120, 0x9, 0x9, 0x14, 0x7f0709c035b8, 0x18)
	bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x2132bc0, 0xc000ba2ba0, 0xc000710120, 0x9, 0x9, 0x9, 0xacfcc5, 0xc000d9e09c, 0xc000f0c000)
	io/io.go:328 +0x87
io.ReadFull(...)
	io/io.go:347
golang.org/x/net/http2.readFrameHeader(0xc000710120, 0x9, 0x9, 0x2132bc0, 0xc000ba2ba0, 0x0, 0x0, 0x215b160, 0xc000d9e090)
	mcclainmichael@example.org/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0007100e0, 0xc000d9e090, 0x2f5b260, 0x0, 0x0)
	mcclainmichael@example.org/http2/frame.go:498 +0xa5
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0008a3b00, 0xc0006ba600, 0x1edec20)
	mcclainmichael@example.org/internal/transport/http2_server.go:599 +0x9b
google.golang.org/grpc.(*Server).serveStreams(0xc000bbb180, 0x21890f8, 0xc0008a3b00)
	mcclainmichael@example.org/server.go:905 +0x103
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000bbb180, 0x21890f8, 0xc0008a3b00, 0xc000d9ec78, 0x13)
	mcclainmichael@example.org/server.go:847 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	mcclainmichael@example.org/server.go:846 +0x190

goroutine 180 [select, 657 minutes]:
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).Stream(0xc000612b00, 0x7f06e2e7ccb8, 0xc0008121a0, 0x7f06e2e7ccb8, 0xc0008121a0)
	istio.io/istio/pilot/pkg/xds/ads.go:321 +0x77f
istio.io/istio/security/pkg/nodeagent/sds.(*sdsservice).StreamSecrets(0xc000614c30, 0x21855e0, 0xc0008121a0, 0x2171670, 0xc000614c30)
	istio.io/istio/security/pkg/nodeagent/sds/sdsservice.go:189 +0x75
github.com/envoyproxy/go-control-plane/envoy/service/secret/v3._SecretDiscoveryService_StreamSecrets_Handler(0x1c93c00, 0xc000614c30, 0x217e320, 0xc000298480, 0x1a199a0, 0xc000812190)
	mcclainmichael@example.org/envoy/service/secret/v3/sds.pb.go:355 +0xad
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000bbb180, 0x21890f8, 0xc0008a3b00, 0xc0007c0240, 0xc000deef00, 0x2f34540, 0x0, 0x0, 0x0)
	mcclainmichael@example.org/server.go:1540 +0xee5
google.golang.org/grpc.(*Server).handleStream(0xc000bbb180, 0x21890f8, 0xc0008a3b00, 0xc0007c0240, 0x0)
	mcclainmichael@example.org/server.go:1620 +0xca5
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00055a040, 0xc000bbb180, 0x21890f8, 0xc0008a3b00, 0xc0007c0240)
	mcclainmichael@example.org/server.go:921 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	mcclainmichael@example.org/server.go:919 +0x1fd

goroutine 151 [select, 657 minutes]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000802190, 0xc000a5af70, 0x5, 0x5, 0x218df00, 0xc000c9c180, 0xc001020200)
	mcclainmichael@example.org/internal/transport/transport.go:178 +0xa5
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000802190, 0xc000a5af70, 0x5, 0x5, 0x218df00, 0xc000d9e000, 0x1ee1698)
	mcclainmichael@example.org/internal/transport/transport.go:172 +0x211
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0006baa50, 0xc000a5af70, 0x5, 0x5, 0xc0002695f0, 0xc0005c7a50, 0xd6009c)
	mcclainmichael@example.org/internal/transport/transport.go:484 +0x55
io.ReadAtLeast(0x213efc0, 0xc0006baa50, 0xc000a5af70, 0x5, 0x5, 0x5, 0x0, 0x218df00, 0xc000c9c180)
	io/io.go:328 +0x87
io.ReadFull(...)
	io/io.go:347
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0007c0000, 0xc000a5af70, 0x5, 0x5, 0x218df00, 0xc000c9c180, 0xc001020200)
	mcclainmichael@example.org/internal/transport/transport.go:468 +0xd2
google.golang.org/grpc.(*parser).recvMsg(0xc000a5af60, 0x400000, 0x0, 0xc0005c7bb8, 0xd672ce, 0xc001020200, 0x75, 0x75)
	mcclainmichael@example.org/rpc_util.go:559 +0x63
google.golang.org/grpc.recvAndDecompress(0xc000a5af60, 0xc0007c0000, 0x0, 0x0, 0x400000, 0x0, 0x0, 0x0, 0x418dd3, 0x7f06e2e750e8, ...)
	mcclainmichael@example.org/rpc_util.go:690 +0x4d
google.golang.org/grpc.recv(0xc000a5af60, 0x7f06e2b609d8, 0x2f8b100, 0xc0007c0000, 0x0, 0x0, 0x1d6b7c0, 0xc0006f3e00, 0x400000, 0x0, ...)
	mcclainmichael@example.org/rpc_util.go:758 +0x9b
google.golang.org/grpc.(*serverStream).RecvMsg(0xc00047c900, 0x1d6b7c0, 0xc0006f3e00, 0x0, 0x0)
	mcclainmichael@example.org/stream.go:1563 +0x156
github.com/envoyproxy/go-control-plane/envoy/service/secret/v3.(*secretDiscoveryServiceStreamSecretsServer).Recv(0xc00055f140, 0xc0005c7e70, 0x0, 0x1)
	mcclainmichael@example.org/envoy/service/secret/v3/sds.pb.go:374 +0x62
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).receive(0xc000612b00, 0xc00047c9c0)
	istio.io/istio/pilot/pkg/xds/ads.go:157 +0xab
created by istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).Stream
	istio.io/istio/pilot/pkg/xds/ads.go:311 +0x6ab

goroutine 77 [select, 8577 minutes]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000802230, 0xc00088a298, 0x5, 0x5, 0x218df00, 0xc000ad2000, 0xc0007d2080)
	mcclainmichael@example.org/internal/transport/transport.go:178 +0xa5
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000802230, 0xc00088a298, 0x5, 0x5, 0x218df00, 0xc000d9e018, 0x1ee1698)
	mcclainmichael@example.org/internal/transport/transport.go:172 +0x211
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0006bac60, 0xc00088a298, 0x5, 0x5, 0xc0002695f0, 0xc000ba1a50, 0xd6009c)
	mcclainmichael@example.org/internal/transport/transport.go:484 +0x55
io.ReadAtLeast(0x213efc0, 0xc0006bac60, 0xc00088a298, 0x5, 0x5, 0x5, 0x0, 0x218df00, 0xc000ad2000)
	io/io.go:328 +0x87
io.ReadFull(...)
	io/io.go:347
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0007c0240, 0xc00088a298, 0x5, 0x5, 0x218df00, 0xc000ad2000, 0xc0007d2080)
	mcclainmichael@example.org/internal/transport/transport.go:468 +0xd2
google.golang.org/grpc.(*parser).recvMsg(0xc00088a288, 0x400000, 0x0, 0xc000ba1bb8, 0xd672ce, 0xc0007d2080, 0x74, 0x74)
	mcclainmichael@example.org/rpc_util.go:559 +0x63
google.golang.org/grpc.recvAndDecompress(0xc00088a288, 0xc0007c0240, 0x0, 0x0, 0x400000, 0x0, 0x0, 0x0, 0x418dd3, 0x7f06e2e73ba8, ...)
	mcclainmichael@example.org/rpc_util.go:690 +0x4d
google.golang.org/grpc.recv(0xc00088a288, 0x7f06e2b609d8, 0x2f8b100, 0xc0007c0240, 0x0, 0x0, 0x1d6b7c0, 0xc00060a000, 0x400000, 0x0, ...)
	mcclainmichael@example.org/rpc_util.go:758 +0x9b
google.golang.org/grpc.(*serverStream).RecvMsg(0xc000298480, 0x1d6b7c0, 0xc00060a000, 0x0, 0x0)
	mcclainmichael@example.org/stream.go:1563 +0x156
github.com/envoyproxy/go-control-plane/envoy/service/secret/v3.(*secretDiscoveryServiceStreamSecretsServer).Recv(0xc0008121a0, 0xc000ba1e70, 0x0, 0x1)
	mcclainmichael@example.org/envoy/service/secret/v3/sds.pb.go:374 +0x62
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).receive(0xc000612b00, 0xc000c62000)
	istio.io/istio/pilot/pkg/xds/ads.go:157 +0xab
created by istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).Stream
	istio.io/istio/pilot/pkg/xds/ads.go:311 +0x6ab

goroutine 181 [IO wait]:
internal/poll.runtime_pollWait(0x7f06e2f5c4c8, 0x72, 0xffffffffffffffff)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000176e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000176e00, 0xc000a7e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000176e00, 0xc000a7e000, 0x1000, 0x1000, 0x2172300, 0xc000b3c750, 0xc000b8d740)
	net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000c840e0, 0xc000a7e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	net/net.go:183 +0x91
net/http.(*connReader).Read(0xc000e681b0, 0xc000a7e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	net/http/server.go:780 +0x1b9
bufio.(*Reader).fill(0xc000ba2120)
	bufio/bufio.go:101 +0x108
bufio.(*Reader).ReadSlice(0xc000ba2120, 0xc000b3c70a, 0x0, 0x0, 0xc000b3c750, 0x4aed96, 0xc00095d5cb)
	bufio/bufio.go:360 +0x3d
bufio.(*Reader).ReadLine(0xc000ba2120, 0x203000, 0x203000, 0x2, 0x0, 0x0, 0xc000b8d8f8)
	bufio/bufio.go:389 +0x34
net/textproto.(*Reader).readLineSlice(0xc000614540, 0xc000a9b000, 0xc000b8d9e8, 0x4e2cb3, 0xc000176e00, 0xc000a7f000)
	net/textproto/reader.go:57 +0xd6
net/textproto.(*Reader).ReadLine(...)
	net/textproto/reader.go:38
net/http.readRequest(0xc000ba2120, 0x0, 0xc000a9b000, 0x0, 0x0)
	net/http/request.go:1027 +0xaa
net/http.(*conn).readRequest(0xc000553e00, 0x2175980, 0xc000a62c80, 0x0, 0x0, 0x0)
	net/http/server.go:966 +0x19d
net/http.(*conn).serve(0xc000553e00, 0x2175a28, 0xc000a62c80)
	net/http/server.go:1858 +0x705
created by net/http.(*Server).Serve
	net/http/server.go:2993 +0x39b

goroutine 111347 [select, 5761 minutes]:
google.golang.org/grpc/internal/transport.(*writeQuota).get(0xc0006e2740, 0xc00002829a, 0x0, 0x0)
	mcclainmichael@example.org/internal/transport/flowcontrol.go:59 +0x7a
google.golang.org/grpc/internal/transport.(*http2Server).Write(0xc0009e6480, 0xc0002be120, 0xc000baa040, 0x5, 0x5, 0xc000bc6000, 0x28295, 0x28295, 0xc000baa045, 0x5, ...)
	mcclainmichael@example.org/internal/transport/http2_server.go:1078 +0x136
google.golang.org/grpc.(*serverStream).SendMsg(0xc000c62180, 0x1d6b900, 0xc000ad3000, 0x0, 0x0)
	mcclainmichael@example.org/stream.go:1510 +0x242
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3.(*aggregatedDiscoveryServiceStreamAggregatedResourcesServer).Send(0xc000e90220, 0xc000ad3000, 0x1ca7380, 0xc0009eede0)
	mcclainmichael@example.org/envoy/service/discovery/v3/ads.pb.go:312 +0x49
istio.io/istio/pkg/istio-agent.sendDownstream.func1(0x20, 0x20)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:741 +0x38
istio.io/istio/pilot/pkg/grpc.Send(0x2175a28, 0xc000e4e900, 0xc000d4e040, 0x0, 0x0)
	istio.io/istio/pilot/pkg/grpc/grpc.go:53 +0x24a
istio.io/istio/pkg/istio-agent.sendDownstream(0x7f06e2f28160, 0xc000e90220, 0xc000ad3000, 0x2175a01, 0xc00089a3f0)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:741 +0xa9
istio.io/istio/pkg/istio-agent.forwardToEnvoy(0xc000ab6800, 0xc000ad3000)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:559 +0x73
istio.io/istio/pkg/istio-agent.(*XdsProxy).handleUpstreamResponse(0xc000aff760, 0xc000ab6800)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:518 +0x35e
created by istio.io/istio/pkg/istio-agent.(*XdsProxy).HandleUpstream
	istio.io/istio/pkg/istio-agent/xds_proxy.go:419 +0x405

goroutine 164 [IO wait]:
internal/poll.runtime_pollWait(0x7f06e2f5c2f8, 0x72, 0xffffffffffffffff)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000ad2d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000ad2d80, 0xc000228000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000ad2d80, 0xc000228000, 0x1000, 0x1000, 0x2172300, 0xc000df4030, 0xc000d6e740)
	net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00000e0d8, 0xc000228000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	net/net.go:183 +0x91
net/http.(*connReader).Read(0xc000b98390, 0xc000228000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	net/http/server.go:780 +0x1b9
bufio.(*Reader).fill(0xc000750720)
	bufio/bufio.go:101 +0x108
bufio.(*Reader).ReadSlice(0xc000750720, 0xc000df400a, 0x0, 0x0, 0xc000df4030, 0x4aed96, 0xc0007107cb)
	bufio/bufio.go:360 +0x3d
bufio.(*Reader).ReadLine(0xc000750720, 0x203000, 0x203000, 0x2, 0x0, 0x0, 0xc000d6e8f8)
	bufio/bufio.go:389 +0x34
net/textproto.(*Reader).readLineSlice(0xc000e98a50, 0xc0000e7b00, 0xc000d6e9e8, 0x4e2cb3, 0xc000ad2d80, 0xc00065e000)
	net/textproto/reader.go:57 +0xd6
net/textproto.(*Reader).ReadLine(...)
	net/textproto/reader.go:38
net/http.readRequest(0xc000750720, 0x0, 0xc0000e7b00, 0x0, 0x0)
	net/http/request.go:1027 +0xaa
net/http.(*conn).readRequest(0xc00004ca00, 0x2175980, 0xc0006dd080, 0x0, 0x0, 0x0)
	net/http/server.go:966 +0x19d
net/http.(*conn).serve(0xc00004ca00, 0x2175a28, 0xc0006dd080)
	net/http/server.go:1858 +0x705
created by net/http.(*Server).Serve
	net/http/server.go:2993 +0x39b

goroutine 111310 [select, 5757 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000e74b90)
	mcclainmichael@example.org/balancer_conn_wrappers.go:77 +0xb2
created by google.golang.org/grpc.newCCBalancerWrapper
	mcclainmichael@example.org/balancer_conn_wrappers.go:67 +0x1e6

goroutine 111309 [select, 5761 minutes]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000e749b0, 0xc000ae2358, 0x5, 0x5, 0x218df00, 0xc000176d00, 0xc000892820)
	mcclainmichael@example.org/internal/transport/transport.go:178 +0xa5
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000e749b0, 0xc000ae2358, 0x5, 0x5, 0x218df00, 0xc000766240, 0x1ee1698)
	mcclainmichael@example.org/internal/transport/transport.go:172 +0x211
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000e4e780, 0xc000ae2358, 0x5, 0x5, 0xc0002695f0, 0xc00096db50, 0xd6009c)
	mcclainmichael@example.org/internal/transport/transport.go:484 +0x55
io.ReadAtLeast(0x213efc0, 0xc000e4e780, 0xc000ae2358, 0x5, 0x5, 0x5, 0x0, 0x218df00, 0xc000176d00)
	io/io.go:328 +0x87
io.ReadFull(...)
	io/io.go:347
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0002be120, 0xc000ae2358, 0x5, 0x5, 0x218df00, 0xc000176d00, 0xc000892820)
	mcclainmichael@example.org/internal/transport/transport.go:468 +0xd2
google.golang.org/grpc.(*parser).recvMsg(0xc000ae2348, 0x400000, 0x0, 0xc00096dcb8, 0xd672ce, 0xc000892820, 0x9a, 0x9a)
	mcclainmichael@example.org/rpc_util.go:559 +0x63
google.golang.org/grpc.recvAndDecompress(0xc000ae2348, 0xc0002be120, 0x0, 0x0, 0x400000, 0x0, 0x0, 0x0, 0x1, 0x0, ...)
	mcclainmichael@example.org/rpc_util.go:690 +0x4d
google.golang.org/grpc.recv(0xc000ae2348, 0x7f06e2b609d8, 0x2f8b100, 0xc0002be120, 0x0, 0x0, 0x1d6b7c0, 0xc000176e80, 0x400000, 0x0, ...)
	mcclainmichael@example.org/rpc_util.go:758 +0x9b
google.golang.org/grpc.(*serverStream).RecvMsg(0xc000c62180, 0x1d6b7c0, 0xc000176e80, 0x0, 0x0)
	mcclainmichael@example.org/stream.go:1563 +0x156
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3.(*aggregatedDiscoveryServiceStreamAggregatedResourcesServer).Recv(0xc000e90220, 0xc000176d00, 0x0, 0x0)
	mcclainmichael@example.org/envoy/service/discovery/v3/ads.pb.go:317 +0x62
istio.io/istio/pkg/istio-agent.(*XdsProxy).handleStream.func1(0x7f06e2f28160, 0xc000e90220, 0xc000ab6800, 0xc000a42558, 0xc000aff760)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:332 +0x51
created by istio.io/istio/pkg/istio-agent.(*XdsProxy).handleStream
	istio.io/istio/pkg/istio-agent/xds_proxy.go:329 +0x27b

goroutine 111308 [select, 5783 minutes]:
istio.io/istio/pkg/istio-agent.(*XdsProxy).HandleUpstream(0xc000aff760, 0x2175a28, 0xc000e4eab0, 0xc000ab6800, 0x215a080, 0xc000e90620, 0x0, 0x0)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:422 +0x4aa
istio.io/istio/pkg/istio-agent.(*XdsProxy).handleStream(0xc000aff760, 0x7f06e2f28160, 0xc000e90220, 0x0, 0x0)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:383 +0x58a
istio.io/istio/pkg/istio-agent.(*XdsProxy).StreamAggregatedResources(0xc000aff760, 0x2185530, 0xc000e90220, 0x215bb88, 0xc000aff760)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:310 +0xb3
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3._AggregatedDiscoveryService_StreamAggregatedResources_Handler(0x1def7c0, 0xc000aff760, 0x217e320, 0xc000c62180, 0x1a199a0, 0xc000e90210)
	mcclainmichael@example.org/envoy/service/discovery/v3/ads.pb.go:298 +0xad
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000bbb340, 0x21890f8, 0xc0009e6480, 0xc0002be120, 0xc000def1d0, 0x2f344e0, 0x0, 0x0, 0x0)
	mcclainmichael@example.org/server.go:1540 +0xee5
google.golang.org/grpc.(*Server).handleStream(0xc000bbb340, 0x21890f8, 0xc0009e6480, 0xc0002be120, 0x0)
	mcclainmichael@example.org/server.go:1620 +0xca5
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000a424a0, 0xc000bbb340, 0x21890f8, 0xc0009e6480, 0xc0002be120)
	mcclainmichael@example.org/server.go:921 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	mcclainmichael@example.org/server.go:919 +0x1fd

goroutine 284916 [IO wait]:
internal/poll.runtime_pollWait(0x7f06e2f5c3e0, 0x72, 0xffffffffffffffff)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000066418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000066400, 0xc000224000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000066400, 0xc000224000, 0x1000, 0x1000, 0x43b21c, 0xc000a6cc38, 0x469d20)
	net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000c84000, 0xc000224000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	net/net.go:183 +0x91
net/http.(*persistConn).Read(0xc000ac6c60, 0xc000224000, 0x1000, 0x1000, 0xc000ace0c0, 0xc000a6cd40, 0x405755)
	net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc000ba2480)
	bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc000ba2480, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)
	bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc000ac6c60)
	net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
	net/http/transport.go:1743 +0xc77

goroutine 111306 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0009e6480)
	mcclainmichael@example.org/internal/transport/http2_server.go:1119 +0x265
created by google.golang.org/grpc/internal/transport.NewServerTransport
	mcclainmichael@example.org/internal/transport/http2_server.go:331 +0x1295

goroutine 111346 [select, 5761 minutes]:
istio.io/istio/pkg/istio-agent.(*XdsProxy).handleUpstreamRequest(0xc000aff760, 0xc000ab6800)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:454 +0xe6
created by istio.io/istio/pkg/istio-agent.(*XdsProxy).HandleUpstream
	istio.io/istio/pkg/istio-agent/xds_proxy.go:418 +0x3d1

goroutine 111305 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000e74870, 0x1, 0x0, 0x0, 0x0, 0x0)
	mcclainmichael@example.org/internal/transport/controlbuf.go:407 +0xff
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000486780, 0x0, 0x0)
	mcclainmichael@example.org/internal/transport/controlbuf.go:534 +0x1dd
google.golang.org/grpc/internal/transport.NewServerTransport.func2(0xc0009e6480)
	mcclainmichael@example.org/internal/transport/http2_server.go:322 +0xd7
created by google.golang.org/grpc/internal/transport.NewServerTransport
	mcclainmichael@example.org/internal/transport/http2_server.go:319 +0x1270

goroutine 284917 [select]:
net/http.(*persistConn).writeLoop(0xc000ac6c60)
	net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
	net/http/transport.go:1744 +0xc9c

goroutine 111296 [select, 5783 minutes]:
google.golang.org/grpc.newClientStreamWithParams.func4(0xc0006ac500, 0xc0008ca700, 0x2175980, 0xc000e80380)
	mcclainmichael@example.org/stream.go:333 +0xaa
created by google.golang.org/grpc.newClientStreamWithParams
	mcclainmichael@example.org/stream.go:332 +0x749

goroutine 111345 [select, 5761 minutes]:
istio.io/istio/pkg/istio-agent.(*XdsProxy).HandleUpstream.func1(0xc000ab6800)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:411 +0x109
created by istio.io/istio/pkg/istio-agent.(*XdsProxy).HandleUpstream
	istio.io/istio/pkg/istio-agent/xds_proxy.go:400 +0x39f

goroutine 285091 [IO wait]:
internal/poll.runtime_pollWait(0x7f06e2f5c5b0, 0x72, 0xffffffffffffffff)
	runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0008ac098, 0x72, 0x0, 0x1, 0xffffffffffffffff)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0008ac080, 0xc000e99151, 0x1, 0x1, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0008ac080, 0xc000e99151, 0x1, 0x1, 0xc00072c4e0, 0x0, 0xc000a62d18)
	net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00068edb8, 0xc000e99151, 0x1, 0x1, 0x0, 0x0, 0x0)
	net/net.go:183 +0x91
net/http.(*connReader).backgroundRead(0xc000e99140)
	net/http/server.go:672 +0x58
created by net/http.(*connReader).startBackgroundRead
	net/http/server.go:668 +0xd5
```

</details>

Though the logs comes from istio 1.12, I think it may also apply to istio 1.14.

The interesting part is :

```
goroutine 111309 [select, 5761 minutes]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000e749b0, 0xc000ae2358, 0x5, 0x5, 0x218df00, 0xc000176d00, 0xc000892820)
	mcclainmichael@example.org/internal/transport/transport.go:178 +0xa5
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000e749b0, 0xc000ae2358, 0x5, 0x5, 0x218df00, 0xc000766240, 0x1ee1698)
	mcclainmichael@example.org/internal/transport/transport.go:172 +0x211
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000e4e780, 0xc000ae2358, 0x5, 0x5, 0xc0002695f0, 0xc00096db50, 0xd6009c)
	mcclainmichael@example.org/internal/transport/transport.go:484 +0x55
io.ReadAtLeast(0x213efc0, 0xc000e4e780, 0xc000ae2358, 0x5, 0x5, 0x5, 0x0, 0x218df00, 0xc000176d00)
	io/io.go:328 +0x87
io.ReadFull(...)
	io/io.go:347
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0002be120, 0xc000ae2358, 0x5, 0x5, 0x218df00, 0xc000176d00, 0xc000892820)
	mcclainmichael@example.org/internal/transport/transport.go:468 +0xd2
google.golang.org/grpc.(*parser).recvMsg(0xc000ae2348, 0x400000, 0x0, 0xc00096dcb8, 0xd672ce, 0xc000892820, 0x9a, 0x9a)
	mcclainmichael@example.org/rpc_util.go:559 +0x63
google.golang.org/grpc.recvAndDecompress(0xc000ae2348, 0xc0002be120, 0x0, 0x0, 0x400000, 0x0, 0x0, 0x0, 0x1, 0x0, ...)
	mcclainmichael@example.org/rpc_util.go:690 +0x4d
google.golang.org/grpc.recv(0xc000ae2348, 0x7f06e2b609d8, 0x2f8b100, 0xc0002be120, 0x0, 0x0, 0x1d6b7c0, 0xc000176e80, 0x400000, 0x0, ...)
	mcclainmichael@example.org/rpc_util.go:758 +0x9b
google.golang.org/grpc.(*serverStream).RecvMsg(0xc000c62180, 0x1d6b7c0, 0xc000176e80, 0x0, 0x0)
	mcclainmichael@example.org/stream.go:1563 +0x156
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3.(*aggregatedDiscoveryServiceStreamAggregatedResourcesServer).Recv(0xc000e90220, 0xc000176d00, 0x0, 0x0)
	mcclainmichael@example.org/envoy/service/discovery/v3/ads.pb.go:317 +0x62
istio.io/istio/pkg/istio-agent.(*XdsProxy).handleStream.func1(0x7f06e2f28160, 0xc000e90220, 0xc000ab6800, 0xc000a42558, 0xc000aff760)
	istio.io/istio/pkg/istio-agent/xds_proxy.go:332 +0x51
created by istio.io/istio/pkg/istio-agent.(*XdsProxy).handleStream
	istio.io/istio/pkg/istio-agent/xds_proxy.go:329 +0x27b
```


 Comment 3: So it's caused by #39209 ?
 Comment 4: Looks similar, you can test with new release
 Comment 5: @hzxuzhonghu Thanks, we are now running modified 1.12 and 1.14 (custom build, with backported #39309 #39310)

{'source': 'GithubIssues', 'repo_name_with_owner': 'istio/istio', 'issue_id': 1304198617, 'repo_stars': 32221.0}