title: Prometheus is crash looping when the pod recreates
body: <p>We are <strong>Prometheus version 2.26.0</strong> and <strong>kubernetes verion 1.21.7 in Azure</strong>. We mount the data in <strong>Azure storage NFS</strong> and it was working fine. From last few days Prometheus container is crash-looping and below are the logs</p>
  <pre><code>level=info ts=2022-01-26T08:04:14.375Z caller=main.go:418 msg=&quot;Starting Prometheus&quot; version=&quot;(version=2.26.0, branch=HEAD, revision=3cafc58827d1ebd1a67749f88be4218f0bab3d8d)&quot;
  level=info ts=2022-01-26T08:04:14.375Z caller=main.go:423 build_context=&quot;(go=go1.16.2, user=root@a67cafebe6d0, date=20210331-11:56:23)&quot;
  level=info ts=2022-01-26T08:04:14.375Z caller=main.go:424 host_details=&quot;(Linux 5.4.0-1065-azure #68~18.04.1-Ubuntu SMP Fri Dec 3 14:08:44 UTC 2021 x86_64 prometheus-6b9d9d54f4-nc45x (none))&quot;
  level=info ts=2022-01-26T08:04:14.375Z caller=main.go:425 fd_limits=&quot;(soft=1048576, hard=1048576)&quot;
  level=info ts=2022-01-26T08:04:14.375Z caller=main.go:426 vm_limits=&quot;(soft=unlimited, hard=unlimited)&quot;
  level=info ts=2022-01-26T08:04:14.503Z caller=web.go:540 component=web msg=&quot;Start listening for connections&quot; address=0.0.0.0:9090
  level=info ts=2022-01-26T08:04:14.507Z caller=main.go:795 msg=&quot;Starting TSDB ...&quot;
  level=info ts=2022-01-26T08:04:14.509Z caller=tls_config.go:191 component=web msg=&quot;TLS is disabled.&quot; http2=false
  level=info ts=2022-01-26T08:04:14.560Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641478251052 maxt=1641513600000 ulid=01FRSEHC4YHV3N26JY5AMNZFRW
  level=info ts=2022-01-26T08:04:14.593Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641513600037 maxt=1641578400000 ulid=01FRVCAP2VJGDF0Z9CS24EXAJJ
  level=info ts=2022-01-26T08:04:14.624Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641578400038 maxt=1641643200000 ulid=01FRXA4AQHMHAEYWRKQFGP075M
  level=info ts=2022-01-26T08:04:14.651Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641643200422 maxt=1641708000000 ulid=01FRZ7XQQ4RA96DCPPBP22D71N
  level=info ts=2022-01-26T08:04:14.679Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641708000020 maxt=1641772800000 ulid=01FS15QDG6BS7H6M6Y09HG3E12
  level=info ts=2022-01-26T08:04:14.707Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641772800011 maxt=1641837600000 ulid=01FS33GT38PRSB9VP56YFXT2M0
  level=info ts=2022-01-26T08:04:14.736Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641963555381 maxt=1641967200000 ulid=01FS6MRNZEWT1Z6P697K09KHD7
  level=info ts=2022-01-26T08:04:14.763Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641837600100 maxt=1641902400000 ulid=01FS6R88C70TCD8CYC4XJ95X23
  level=info ts=2022-01-26T08:04:14.810Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641967200019 maxt=1642032000000 ulid=01FS8WXQP3YJ7EXBVNYBQG4DVY
  level=info ts=2022-01-26T08:04:14.836Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642032000072 maxt=1642096800000 ulid=01FSATQBR4XBQRDM72ATFS9PQ2
  level=info ts=2022-01-26T08:04:14.863Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642096800059 maxt=1642161600000 ulid=01FSCRHE2YBDX7GPRPSH6BNGRX
  level=info ts=2022-01-26T08:04:14.895Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642161600091 maxt=1642226400000 ulid=01FSEPB1GPGAANVCQ2VKW9BQ4G
  level=info ts=2022-01-26T08:04:14.948Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642226400026 maxt=1642291200000 ulid=01FSGM4J0G1D0A6H1GD3N9C372
  level=info ts=2022-01-26T08:04:14.973Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642291200005 maxt=1642356000000 ulid=01FSJHY6W0FRYDHCXBVB5XPFYG
  level=info ts=2022-01-26T08:04:15.002Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642356000027 maxt=1642420800000 ulid=01FSMFR96DASV6YPN66W7C86H9
  level=info ts=2022-01-26T08:04:15.077Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642420800042 maxt=1642485600000 ulid=01FSPDHGWRT65D8CKWQ2JPRHW3
  level=info ts=2022-01-26T08:04:15.105Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642485600006 maxt=1642550400000 ulid=01FSRBAVP2MW71H08F32D6HGB4
  level=info ts=2022-01-26T08:04:15.130Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642550400028 maxt=1642615200000 ulid=01FST9482FD0Z3PHXHNW2W616E
  level=info ts=2022-01-26T08:04:15.157Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642680000018 maxt=1642687200000 ulid=01FSW00TJKJ7CGCQ7JJS3XQK8G
  level=info ts=2022-01-26T08:04:15.187Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642687200018 maxt=1642694400000 ulid=01FSW6WHTSEAXHWV5J7PQP94X7
  level=info ts=2022-01-26T08:04:15.213Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642615200021 maxt=1642680000000 ulid=01FSW6XYH2Y429PG5YRM0K45XS
  level=info ts=2022-01-26T08:04:15.275Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642694400018 maxt=1642701600000 ulid=01FSWDR92Y7H302NDZRX1V2PX9
  level=info ts=2022-01-26T08:04:21.840Z caller=head.go:696 component=tsdb msg=&quot;Replaying on-disk memory mappable chunks if any&quot;
  level=info ts=2022-01-26T08:04:22.623Z caller=head.go:710 component=tsdb msg=&quot;On-disk memory mappable chunks replay completed&quot; duration=782.403397ms
  level=info ts=2022-01-26T08:04:22.623Z caller=head.go:716 component=tsdb msg=&quot;Replaying WAL, this may take a while&quot;
  level=info ts=2022-01-26T08:04:34.169Z caller=head.go:742 component=tsdb msg=&quot;WAL checkpoint loaded&quot;
  level=info ts=2022-01-26T08:04:38.895Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=299 maxSegment=7511
  level=warn ts=2022-01-26T08:04:46.423Z caller=main.go:645 msg=&quot;Received SIGTERM, exiting gracefully...&quot;
  level=info ts=2022-01-26T08:04:46.424Z caller=main.go:668 msg=&quot;Stopping scrape discovery manager...&quot;
  level=info ts=2022-01-26T08:04:46.424Z caller=main.go:682 msg=&quot;Stopping notify discovery manager...&quot;
  level=info ts=2022-01-26T08:04:46.424Z caller=main.go:704 msg=&quot;Stopping scrape manager...&quot;
  level=info ts=2022-01-26T08:04:46.424Z caller=main.go:678 msg=&quot;Notify discovery manager stopped&quot;
  level=info ts=2022-01-26T08:04:46.425Z caller=main.go:698 msg=&quot;Scrape manager stopped&quot;
  level=info ts=2022-01-26T08:04:46.426Z caller=manager.go:934 component=&quot;rule manager&quot; msg=&quot;Stopping rule manager...&quot;
  level=info ts=2022-01-26T08:04:46.426Z caller=manager.go:944 component=&quot;rule manager&quot; msg=&quot;Rule manager stopped&quot;
  level=info ts=2022-01-26T08:04:46.426Z caller=notifier.go:601 component=notifier msg=&quot;Stopping notification manager...&quot;
  level=info ts=2022-01-26T08:04:46.426Z caller=main.go:872 msg=&quot;Notifier manager stopped&quot;
  level=info ts=2022-01-26T08:04:46.426Z caller=main.go:664 msg=&quot;Scrape discovery manager stopped&quot;
  level=info ts=2022-01-26T08:04:46.792Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=300 maxSegment=7511
  level=info ts=2022-01-26T08:04:46.870Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=301 maxSegment=7511
  level=info ts=2022-01-26T08:04:46.901Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=302 maxSegment=7511
  level=info ts=2022-01-26T08:04:46.946Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=303 maxSegment=7511
  level=info ts=2022-01-26T08:04:46.974Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=304 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.008Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=305 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.034Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=306 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.067Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=307 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.098Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=308 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.124Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=309 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.158Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=310 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.203Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=311 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.254Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=312 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.486Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=313 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.511Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=314 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.539Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=315 maxSegment=7511
  level=info ts=2022-01-26T08:04:47.564Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=316 maxSegment=7511
  
  .
  .
  .
  .
  .
  .
  .
  .
  .
  
  level=info ts=2022-01-26T08:05:15.161Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1401 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.182Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1402 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.205Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1403 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.229Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1404 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.251Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1405 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.274Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1406 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.297Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1407 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.323Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1408 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.349Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1409 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.372Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1410 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.426Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1411 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.452Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1412 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.475Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1413 maxSegment=7511
  level=info ts=2022-01-26T08:05:15.498Z caller=head.go:768 component=tsdb msg=&quot;WAL segment loaded&quot; segment=1414 maxSegment=7511
  rpc error: code = NotFound desc = an error occurred when try to find container &quot;ae14079418f59b04bb80d8413e8fdc34f167bfe762317ef674e05466d34c9e1f&quot;: not found
  </code></pre>
  <p>So I deleted the deployment and redeployed to the the same storage account, the I got new error</p>
  <pre><code>level=info ts=2022-01-26T11:10:11.530Z caller=main.go:418 msg=&quot;Starting Prometheus&quot; version=&quot;(version=2.26.0, branch=HEAD, revision=3cafc58827d1ebd1a67749f88be4218f0bab3d8d)&quot;
  level=info ts=2022-01-26T11:10:11.534Z caller=main.go:423 build_context=&quot;(go=go1.16.2, user=root@a67cafebe6d0, date=20210331-11:56:23)&quot;
  level=info ts=2022-01-26T11:10:11.535Z caller=main.go:424 host_details=&quot;(Linux 5.4.0-1064-azure #67~18.04.1-Ubuntu SMP Wed Nov 10 11:38:21 UTC 2021 x86_64 prometheus-6b9d9d54f4-wnmzh (none))&quot;
  level=info ts=2022-01-26T11:10:11.536Z caller=main.go:425 fd_limits=&quot;(soft=1048576, hard=1048576)&quot;
  level=info ts=2022-01-26T11:10:11.536Z caller=main.go:426 vm_limits=&quot;(soft=unlimited, hard=unlimited)&quot;
  level=info ts=2022-01-26T11:10:14.168Z caller=web.go:540 component=web msg=&quot;Start listening for connections&quot; address=0.0.0.0:9090
  level=info ts=2022-01-26T11:10:15.385Z caller=main.go:795 msg=&quot;Starting TSDB ...&quot;
  level=info ts=2022-01-26T11:10:16.022Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641837600024 maxt=1641902400000 ulid=01FS51ANKBFTVNRPZ68FGQQ5GA
  level=info ts=2022-01-26T11:10:16.309Z caller=tls_config.go:191 component=web msg=&quot;TLS is disabled.&quot; http2=false
  level=info ts=2022-01-26T11:10:16.494Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641902400005 maxt=1641967200000 ulid=01FS6Z46FGXN932K7D39D9166D
  level=info ts=2022-01-26T11:10:16.806Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1641967200106 maxt=1642032000000 ulid=01FS8WXRJ7Q80FKD4C8EJNR0AD
  level=info ts=2022-01-26T11:10:17.011Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642032000003 maxt=1642096800000 ulid=01FSATQE1VMNR101KRW1X10Q75
  level=info ts=2022-01-26T11:10:17.305Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642096800206 maxt=1642161600000 ulid=01FSCRGVT1E7562SF7EQN12JBM
  level=info ts=2022-01-26T11:10:18.240Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642161600059 maxt=1642226400000 ulid=01FSEPAFP2CX03ANRB7Q1AG514
  level=info ts=2022-01-26T11:10:21.046Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642226400051 maxt=1642291200000 ulid=01FSGM3WT0TKR0XW9BD4QSKPQE
  level=info ts=2022-01-26T11:10:21.422Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642291200113 maxt=1642356000000 ulid=01FSJHXKHMANW0E6FXDXVM265G
  level=info ts=2022-01-26T11:10:22.822Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642356000032 maxt=1642420800000 ulid=01FSMFQ6XJ97VJFKNCYQBVB4DZ
  level=info ts=2022-01-26T11:10:23.536Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642420800021 maxt=1642485600000 ulid=01FSPDGM95FDDV2CDWX93BTDCS
  level=info ts=2022-01-26T11:10:23.880Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642485600072 maxt=1642550400000 ulid=01FSRBA555RWY4QNP4HD9YKRBM
  level=info ts=2022-01-26T11:10:25.021Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642550400031 maxt=1642615200000 ulid=01FST93N3C82K9VS20MKTMGGYC
  level=info ts=2022-01-26T11:10:25.713Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642615200014 maxt=1642680000000 ulid=01FSW6X95FRNSN1XJZ2YK0MXW7
  level=info ts=2022-01-26T11:10:26.634Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642680000012 maxt=1642744800000 ulid=01FSY4PXA7V1XQHHA3MC35JSWQ
  level=info ts=2022-01-26T11:10:27.776Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642744800174 maxt=1642809600000 ulid=01FT02G9XGHPV8GME53ZPMYXE6
  level=info ts=2022-01-26T11:10:28.760Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642809600070 maxt=1642874400000 ulid=01FT209WP8AXXVZB1NCSC55ACE
  level=info ts=2022-01-26T11:10:29.618Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642874400253 maxt=1642939200000 ulid=01FT3Y3A4H72FFW318RKHEXXGA
  level=info ts=2022-01-26T11:10:30.313Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1642939200047 maxt=1643004000000 ulid=01FT5VX3YC838QN5VQFAERV1QX
  level=info ts=2022-01-26T11:10:30.483Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643004000040 maxt=1643068800000 ulid=01FT7SPHC5EV0SS1R0WT04H9FR
  level=info ts=2022-01-26T11:10:30.696Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643068800035 maxt=1643133600000 ulid=01FT9QFZXBZ7EYY2CTE8WXZTB9
  level=info ts=2022-01-26T11:10:31.838Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643133600000 maxt=1643155200000 ulid=01FTA574G4M45WX97Z470DQF73
  level=info ts=2022-01-26T11:10:33.686Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643176800008 maxt=1643184000000 ulid=01FTASSZCG8V5N2VGAGFBYJBSR
  level=info ts=2022-01-26T11:10:36.078Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643184000000 maxt=1643191200000 ulid=01FTB0NP47JW5JCF808QZZ8WZQ
  level=info ts=2022-01-26T11:10:36.442Z caller=repair.go:57 component=tsdb msg=&quot;Found healthy block&quot; mint=1643155200065 maxt=1643176800000 ulid=01FTB0P9H3H09B2ADD5X1RXFW6
  level=info ts=2022-01-26T11:10:40.079Z caller=main.go:668 msg=&quot;Stopping scrape discovery manager...&quot;
  level=info ts=2022-01-26T11:10:40.079Z caller=main.go:682 msg=&quot;Stopping notify discovery manager...&quot;
  level=info ts=2022-01-26T11:10:40.079Z caller=main.go:704 msg=&quot;Stopping scrape manager...&quot;
  level=info ts=2022-01-26T11:10:40.079Z caller=main.go:678 msg=&quot;Notify discovery manager stopped&quot;
  level=info ts=2022-01-26T11:10:40.079Z caller=main.go:664 msg=&quot;Scrape discovery manager stopped&quot;
  level=info ts=2022-01-26T11:10:40.079Z caller=main.go:698 msg=&quot;Scrape manager stopped&quot;
  level=info ts=2022-01-26T11:10:40.080Z caller=manager.go:934 component=&quot;rule manager&quot; msg=&quot;Stopping rule manager...&quot;
  level=info ts=2022-01-26T11:10:40.080Z caller=manager.go:944 component=&quot;rule manager&quot; msg=&quot;Rule manager stopped&quot;
  level=info ts=2022-01-26T11:10:40.080Z caller=notifier.go:601 component=notifier msg=&quot;Stopping notification manager...&quot;
  level=info ts=2022-01-26T11:10:40.080Z caller=main.go:872 msg=&quot;Notifier manager stopped&quot;
  level=error ts=2022-01-26T11:10:40.080Z caller=main.go:881 err=&quot;opening storage failed: lock DB directory: resource temporarily unavailable&quot;
  </code></pre>
  <p>The yaml is provided by the <strong>Istio</strong> .Below is deployment yaml file.</p>
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    labels:
      component: &quot;server&quot;
      app: prometheus
      release: prometheus
      chart: prometheus-14.6.1
      heritage: Helm
    name: prometheus
    namespace: istio-system
  spec:
    selector:
      matchLabels:
        component: &quot;server&quot;
        app: prometheus
        release: prometheus
    replicas: 1
    template:
      metadata:
        labels:
          component: &quot;server&quot;
          app: prometheus
          release: prometheus
          chart: prometheus-14.6.1
          heritage: Helm
          
          sidecar.istio.io/inject: &quot;false&quot;
      spec:
        enableServiceLinks: true
        serviceAccountName: prometheus
        containers:
          - name: prometheus-server-configmap-reload
            image: &quot;jimmidyson/configmap-reload:v0.5.0&quot;
            imagePullPolicy: &quot;IfNotPresent&quot;
            args:
              - --volume-dir=/etc/config
              - --webhook-url=http://127.0.0.1:9090/-/reload
            resources:
              {}
            volumeMounts:
              - name: config-volume
                mountPath: /etc/config
                readOnly: true
  
          - name: prometheus-server
            image: &quot;prom/prometheus:v2.26.0&quot;
            imagePullPolicy: &quot;IfNotPresent&quot;
            args:
              - --storage.tsdb.retention.time=15d
              - --config.file=/etc/config/prometheus.yml
              - --storage.tsdb.path=/data
              - --web.console.libraries=/etc/prometheus/console_libraries
              - --web.console.templates=/etc/prometheus/consoles
              - --web.enable-lifecycle
            ports:
              - containerPort: 9090
            readinessProbe:
              httpGet:
                path: /-/ready
                port: 9090
              initialDelaySeconds: 0
              periodSeconds: 5
              timeoutSeconds: 4
              failureThreshold: 3
              successThreshold: 1
            livenessProbe:
              httpGet:
                path: /-/healthy
                port: 9090
              initialDelaySeconds: 30
              periodSeconds: 15
              timeoutSeconds: 10
              failureThreshold: 3
              successThreshold: 1
            resources:
              {}
            volumeMounts:
              - name: config-volume
                mountPath: /etc/config
              - name: azurefileshare
                mountPath: /data
                subPath: &quot;&quot;
        hostNetwork: false
        dnsPolicy: ClusterFirst
        securityContext:
          fsGroup: 65534
          runAsGroup: 65534
          runAsNonRoot: true
          runAsUser: 65534
        terminationGracePeriodSeconds: 300
        volumes:
          - name: config-volume
            configMap:
              name: prometheus
          - name: azurefileshare
            azureFile:
              secretName: log-storage-secret
              shareName: prometheusfileshare
              readOnly: false
  </code></pre>
  <p><strong>Expected Behavior</strong>
  When I mount the data to new container, It should load the data.</p>
  <p><strong>Actual Behavior</strong>
  Not able to load the data or not able bind the data with newly created pod when old pod dies</p>
  <p>Help me out, to resolve the issue.</p>