title: kubernetes timescaledb statefulset: Changes lost on pod recreation
body: <p>I have a Timescaledb server running as StatefulSet in AKS. It appears when I delete and recreate timescaledb pod, the changes are lost even though the pod is associated to the initially associated PV (persistent volume). Any help is appreciated.</p>
  
  <p>Below is the PV, PVC config of statefulset extracted by running <code>kubectl get statefulset timescaledb -o yaml</code></p>
  
  <pre><code>  template:
      metadata:
        creationTimestamp: null
        labels:
          app: timescaledb
      spec:
        containers:
        - args:
          - -c
          - config_file=/etc/postgresql/postgresql.conf
          env:
          - name: POSTGRES_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: timescaledb-secret
          image: docker.io/timescale/timescaledb:latest-pg9.6
          name: timescaledb-backend
          ports:
          - containerPort: 5432
            name: server
            protocol: TCP
          resources:
            requests:
              cpu: "3"
              memory: 6Gi
          volumeMounts:
          - mountPath: /var/lib/postgresql
            name: timescaledbdata
          - mountPath: /etc/postgresql
            name: timescaledb-config
        volumes:
        - configMap:
            defaultMode: 420
            name: timescaledb-config
          name: timescaledb-config
    volumeClaimTemplates:
    - metadata:
        annotations:
          volume.alpha.kubernetes.io/storage-class: standard
        creationTimestamp: null
        name: timescaledbdata
      spec:
        accessModes:
        - ReadWriteOnce
        dataSource: null
        resources:
          requests:
            storage: 200Gi
      status:
        phase: Pending
  </code></pre>
  
  <p>Below demonstrates a temp DB test_db created is lost after pod recreation and during the whole process, the pod was associated to the same PV/disk on Azure.</p>
  
  <pre><code>root@e70a91715239:~/keys# k get pvc -l app=timescaledb
  NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  timescaledbdata-timescaledb-0   Bound    pvc-c7eb99cf-6a6b-11e9-b661-be660567cc75   200Gi      RWO            default        83d
  
  root@e70a91715239:~/keys# k exec -ti timescaledb-0 bash
  bash-4.4# psql -U postgres;
  psql (9.6.13)
  Type "help" for help.
  
  postgres=# create database test_db;
  CREATE DATABASE
  postgres=# \l
                                   List of databases
     Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
  -----------+----------+----------+------------+------------+-----------------------
   postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
   template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
             |          |          |            |            | postgres=CTc/postgres
   template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
             |          |          |            |            | postgres=CTc/postgres
   test_db   | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
  (4 rows)
  
  
  root@e70a91715239:~/keys# k get pods | grep timescale
  timescaledb-0                         1/1     Running   0          12m
  root@e70a91715239:~/keys# k delete pod/timescaledb-0                            
  pod "timescaledb-0" deleted                                                                                                                                         
  root@e70a91715239:~/keys# k get pods | grep timescale       
  timescaledb-0                         1/1     Running   0          14s   
  
  root@e70a91715239:~/keys# k exec -ti timescaledb-0 bash                                                                                                             
  bash-4.4# psql -U postgres
  psql (9.6.13)
  Type "help" for help.
  
  postgres=# \l
                                   List of databases
     Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges                                                                                    
  -----------+----------+----------+------------+------------+-----------------------                                                                                 
   postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
   template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +                                                                                 
             |          |          |            |            | postgres=CTc/postgres                                                                                  
   template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +                                                                                 
             |          |          |            |            | postgres=CTc/postgres                                                                                  
  (3 rows)
  
  root@e70a91715239:~/keys# k get pvc -l app=timescaledb
  NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  timescaledbdata-timescaledb-0   Bound    pvc-c7eb99cf-6a6b-11e9-b661-be660567cc75   200Gi      RWO            default        83d
  </code></pre>
  
  <p>Possibly it is reinitialising as hinted. Please see <a href="https://siaitesting.blob.core.windows.net/shared/timescaledb.log?sp=r&amp;st=2019-07-22T16:46:31Z&amp;se=2019-08-31T00:46:31Z&amp;spr=https&amp;sv=2018-03-28&amp;sig=gdoyeem8a4CLD6Qk7yLYFsqaYUbJznUthrH1E6XMrmI%3D&amp;sr=b" rel="nofollow noreferrer">logs</a>. Any pointers on why it would do that.</p>
  
  <p>Update 1:
  I had a look at the mounts in the <code>timescale</code> pod, it seems to have different partitions for <code>/var/lib/postgresql</code> and <code>/var/lib/postgresql/data</code>. I do not understand why.</p>
  
  <pre><code>Filesystem                Size      Used Available Use% Mounted on
  overlay                  96.9G     22.1G     74.8G  23% /
  tmpfs                    64.0M         0     64.0M   0% /dev
  tmpfs                     7.8G         0      7.8G   0% /sys/fs/cgroup
  /dev/sda1                96.9G     22.1G     74.8G  23% /docker-entrypoint-initdb.d
  /dev/sda1                96.9G     22.1G     74.8G  23% /dev/termination-log
  shm                      64.0M      4.0K     64.0M   0% /dev/shm
  /dev/sda1                96.9G     22.1G     74.8G  23% /etc/resolv.conf
  /dev/sda1                96.9G     22.1G     74.8G  23% /etc/hostname
  /dev/sda1                96.9G     22.1G     74.8G  23% /etc/hosts
  /dev/sdc                196.7G     59.3M    196.7G   0% /var/lib/postgresql
  /dev/sda1                96.9G     22.1G     74.8G  23% /var/lib/postgresql/data 
  </code></pre>
  
  <p>Do not understand how the above mounts happen for below config</p>
  
  <pre><code>        volumeMounts:
          - mountPath: /var/lib/postgresql
            name: timescaledbdata
          - mountPath: /etc/postgresql
            name: timescaledb-config
  </code></pre>