title: Flink job fails after 10 minutes from initialization
body: <p>I'm having problems with flink application fail.</p>
  
  <p>This streaming job runs shortly after deploying on Yarn.</p>
  
  <p>But is fails after some minutes with below error messages.</p>
  
  <p>Can it be the evidence of high load in low performance yarn cluster?</p>
  
  <ul>
  <li>1.5.0 flink and yarn single job</li>
  <li>Single node is equipped with 100GBytes RAM and 40 v-cores</li>
  <li>48 Yarn node manager.</li>
  <li>2 Kafka topic input ( 150GBytes/hour for each input stream. )</li>
  <li>480 kafka partition.</li>
  <li>10 flink slot per node manager </li>
  </ul>
  
  <p><strong>From the beginning of the flink</strong></p>
  
  <pre>
  Log Type: jobmanager.log
  
  Log Upload Time: Tue Jun 12 18:19:50 +0900 2018
  
  Log Length: 10807897
  
  2018-06-11 18:59:27,167 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
  2018-06-11 18:59:27,168 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @ 14:54:44 UTC)
  2018-06-11 18:59:27,168 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  OS current user: irteam
  2018-06-11 18:59:27,472 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2018-06-11 18:59:27,536 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Current Hadoop/Kerberos user: irteam
  2018-06-11 18:59:27,536 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
  2018-06-11 18:59:27,536 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Maximum heap size: 66667 MiBytes
  2018-06-11 18:59:27,537 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64
  2018-06-11 18:59:27,537 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Hadoop version: 2.8.3
  2018-06-11 18:59:27,537 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM Options:
  2018-06-11 18:59:27,538 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xmx75000m
  2018-06-11 18:59:27,538 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Djava.library.path=/home1/irteam/realtime-tools
  2018-06-11 18:59:27,538 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog.file=/naver/search-cluster/eye/var/logs/application_1528711080009_0002/container_e08_1528711080009_0002_01_000001/jobmanager.log
  2018-06-11 18:59:27,538 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlogback.configurationFile=file:logback.xml
  2018-06-11 18:59:27,538 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog4j.configuration=file:log4j.properties
  2018-06-11 18:59:27,538 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Program Arguments: (none)
  2018-06-11 18:59:27,538 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Class path[omit]
  2018-06-11 18:59:27,539 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
  2018-06-11 18:59:27,539 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Registered UNIX signal handlers for [TERM, HUP, INT]
  2018-06-11 18:59:27,542 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - YARN daemon is running as: irteam Yarn client user obtainer: irteam
  2018-06-11 18:59:27,544 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: env.java.home, "/usr/lib/jvm/java-1.8.0-openjdk"
  2018-06-11 18:59:27,544 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: env.java.opts, "-Djava.library.path=/home1/irteam/realtime-tools"
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.cluster-id, application_1528711080009_0002
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, 0.0.0.0
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 100000
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.network.request-backoff.max, 100000
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: containerized.taskmanager.env.JAVA_HOME, /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: internal.cluster.execution-mode, NORMAL
  2018-06-11 18:59:27,545 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 480
  2018-06-11 18:59:27,546 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 10
  2018-06-11 18:59:27,546 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 100000
  2018-06-11 18:59:27,546 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: containerized.master.env.JAVA_HOME, /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64
  2018-06-11 18:59:27,558 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Setting directories for temporary files to: /home1/irteam/naver/search-cluster/eye/volume/nodemanager/usercache/irteam/appcache/application_1528711080009_0002
  2018-06-11 18:59:27,570 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Starting YarnSessionClusterEntrypoint.
  2018-06-11 18:59:27,570 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install default filesystem.
  2018-06-11 18:59:27,636 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to irteam (auth:SIMPLE)
  2018-06-11 18:59:27,650 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Initializing cluster services.
  2018-06-11 18:59:27,654 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Trying to start actor system at chd004.eye.nfra.io:33524
  2018-06-11 18:59:28,126 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
  2018-06-11 18:59:28,222 INFO  akka.remote.Remoting                                          - Starting remoting
  2018-06-11 18:59:28,322 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@chd004.eye.nfra.io:33524]
  2018-06-11 18:59:28,329 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Actor system started at akka.tcp://flink@chd004.eye.nfra.io:33524
  2018-06-11 18:59:28,348 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /home1/irteam/naver/search-cluster/eye/volume/nodemanager/usercache/irteam/appcache/application_1528711080009_0002/blobStore-c25d4d9d-4ddc-442d-8d5e-7bec36dca006
  2018-06-11 18:59:28,349 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:45733 - max concurrent requests: 50 - max backlog: 1000
  2018-06-11 18:59:28,363 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.
  2018-06-11 18:59:28,367 INFO  org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore  - Initializing FileArchivedExecutionGraphStore: Storage directory /home1/irteam/naver/search-cluster/eye/volume/nodemanager/usercache/irteam/appcache/application_1528711080009_0002/executionGraphStore-63bcf196-410d-4d8c-8388-f270beb53555, expiration time 3600000, maximum cache size 52428800 bytes.
  2018-06-11 18:59:28,388 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Created BLOB cache storage directory /home1/irteam/naver/search-cluster/eye/volume/nodemanager/usercache/irteam/appcache/application_1528711080009_0002/blobStore-02db740f-8c23-46e8-bb24-1f583b6a0b33
  2018-06-11 18:59:28,395 WARN  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Upload directory /tmp/flink-web-8698d702-67fe-437c-b62e-78c2969bf770/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
  2018-06-11 18:59:28,396 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Created directory /tmp/flink-web-8698d702-67fe-437c-b62e-78c2969bf770/flink-web-upload for file uploads.
  2018-06-11 18:59:28,399 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Starting rest endpoint.
  2018-06-11 18:59:28,737 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component log file: /naver/search-cluster/eye/var/logs/application_1528711080009_0002/container_e08_1528711080009_0002_01_000001/jobmanager.log
  2018-06-11 18:59:28,737 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component stdout file: /naver/search-cluster/eye/var/logs/application_1528711080009_0002/container_e08_1528711080009_0002_01_000001/jobmanager.out
  2018-06-11 18:59:28,808 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Rest endpoint listening at chd004.eye.nfra.io:39794
  2018-06-11 18:59:28,808 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - http://chd004.eye.nfra.io:39794 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
  2018-06-11 18:59:28,808 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Web frontend listening at http://chd004.eye.nfra.io:39794.
  2018-06-11 18:59:28,817 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
  2018-06-11 18:59:28,902 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
  2018-06-11 18:59:28,916 INFO  org.apache.flink.yarn.YarnResourceManager                     - ResourceManager akka.tcp://flink@chd004.eye.nfra.io:33524/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
  2018-06-11 18:59:28,917 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting the SlotManager.
  2018-06-11 18:59:29,161 INFO  org.apache.flink.yarn.YarnResourceManager                     - Recovered 0 containers from previous attempts ([]).
  2018-06-11 18:59:29,163 INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy  - yarn.client.max-cached-nodemanagers-proxies : 0
  2018-06-11 18:59:29,174 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Dispatcher akka.tcp://flink@chd004.eye.nfra.io:33524/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000
  2018-06-11 18:59:29,174 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Recovering all persisted jobs.
  2018-06-11 18:59:31,120 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Submitting job 5f090c4f4287db062cee0996da5d5ffc (LCS realtime data).
  2018-06-11 18:59:31,130 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
  2018-06-11 18:59:31,136 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Initializing job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc).
  2018-06-11 18:59:31,144 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=3, delayBetweenRestartAttempts=30000) for LCS realtime data (5f090c4f4287db062cee0996da5d5ffc).
  2018-06-11 18:59:31,148 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/a6ffe322-07db-4282-a29c-0836ad26cd9f .
  2018-06-11 18:59:31,165 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job recovers via failover strategy: full graph restart
  2018-06-11 18:59:31,174 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Running initialization on master for job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc).
  2018-06-11 18:59:31,174 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Successfully ran initialization on master in 0 ms.
  2018-06-11 18:59:31,248 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Using application-defined state backend: File State Backend (checkpoints: 'file:/home1/irteam/apps/flink-1.4.0/checkpoint', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1)
  2018-06-11 18:59:31,248 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Configuring application-defined state backend with job/cluster config
  2018-06-11 18:59:31,258 INFO  org.apache.flink.runtime.jobmaster.JobManagerRunner           - JobManager runner for job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@chd004.eye.nfra.io:33524/user/jobmanager_0.
  2018-06-11 18:59:31,260 INFO  org.apache.flink.runtime.jobmaster.JobMaster                  - Starting execution of job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc)
  2018-06-11 18:59:31,261 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc) switched from state CREATED to RUNNING.
  2018-06-11 18:59:31,264 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source (1/480) (98a01166bb2ac99dd301e4b60febbc45) switched from CREATED to SCHEDULED.
  </pre>
  
  <p><strong>Near the timeout event which might cause flink job fails.</strong></p>
  
  <pre>
  2018-06-12 18:17:39,750 INFO  org.apache.flink.runtime.rest.handler.legacy.backpressure.StackTraceSampleCoordinator  - Cancelling sample 5589
  akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink@chd023.eye.nfra.io:34783/user/taskmanager_0#-297572584]] after [15000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation".
      at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      at java.lang.Thread.run(Thread.java:748)
  2018-06-12 18:17:39,770 INFO  org.apache.flink.runtime.rest.handler.legacy.backpressure.StackTraceSampleCoordinator  - Cancelling sample 5590
  akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink@chd032.eye.nfra.io:34653/user/taskmanager_0#424015125]] after [15000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation".
      at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      at java.lang.Thread.run(Thread.java:748)
  2018-06-12 18:17:51,270 INFO  org.apache.flink.runtime.rest.handler.legacy.backpressure.StackTraceSampleCoordinator  - Cancelling sample 5591
  akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink@chd032.eye.nfra.io:34653/user/taskmanager_0#424015125]] after [15000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation".
      at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      at java.lang.Thread.run(Thread.java:748)
  2018-06-12 18:17:55,650 INFO  org.apache.flink.yarn.YarnResourceManager                     - The heartbeat of TaskManager with id container_e08_1528711080009_0002_01_000017 timed out.
  2018-06-12 18:17:55,650 INFO  org.apache.flink.yarn.YarnResourceManager                     - Closing TaskExecutor connection container_e08_1528711080009_0002_01_000017 because: The heartbeat of TaskManager with id container_e08_1528711080009_0002_01_000017  timed out.
  2018-06-12 18:17:55,650 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Unregister TaskManager 525095d833344e8b205017666accd9c5 from the SlotManager.
  2018-06-12 18:17:55,650 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Window(EventTimeSessionWindows(300000), NowTrigger, NowSessionProcessor) -> Sink: Unnamed (188/480) (f9ed2fc23d6ca5a364300864b60760af) switched from RUNNING to FAILED.
  org.apache.flink.util.FlinkException: Releasing TaskManager container_e08_1528711080009_0002_01_000017.
      at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManagerInternal(SlotPool.java:1067)
      at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManager(SlotPool.java:1050)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
      at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
      at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
      at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
      at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
      at akka.actor.ActorCell.invoke(ActorCell.scala:495)
      at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
      at akka.dispatch.Mailbox.run(Mailbox.scala:224)
      at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  2018-06-12 18:17:55,651 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc) switched from state RUNNING to FAILING.
  org.apache.flink.util.FlinkException: Releasing TaskManager container_e08_1528711080009_0002_01_000017.
      at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManagerInternal(SlotPool.java:1067)
      at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManager(SlotPool.java:1050)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
      at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
      at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
      at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
      at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
      at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
      at akka.actor.ActorCell.invoke(ActorCell.scala:495)
      at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
      at akka.dispatch.Mailbox.run(Mailbox.scala:224)
      at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  2018-06-12 18:17:55,679 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Custom Source (1/480) (98a01166bb2ac99dd301e4b60febbc45) switched from RUNNING to CANCELING.
  </pre>