title: Saving data into Ignite Cluster is not distributed evenly
body: <p>I am working on a use-case to store the data in an in-memory database and enable it to BI for analytics.</p>
  <p>My main objective is,</p>
  <ol>
  <li>Cache data in Ignite, if the data is more than available heap size, spill the data into Disk.</li>
  <li>Upsert the CDC changes to the existing tables.</li>
  <li>Enable Ignite table via JDBC from BI for analytics.</li>
  <li>Analytics BI should be refreshed within 2 to 3secs.</li>
  </ol>
  <p>As a part of this exercise, I was trying to load History data(~700m rows, size is approx 87Gi) into Ignite using Spark.</p>
  <p>I was able to integrate spark with ignite and successfully saved the dataframe into Ignite table. After loading 8M records, I have observed that the table partitions are not distributed evenly in the cluster . Also, the data size on the disk is more compared to the Source data size. Approx 8M records occupied 20Gi space on disk. I tried all configuration settings but did not succeed to distribute the data evenly in cluster and compressed the data file on desk. Can someone pls help me fix the configuration issue? Did I miss any Ignite &amp; Spark configuration to distribute the data evenly? Also how to check how much data is cached?</p>
  <p>Please let me know if I need to provide more information.</p>
  <p>Ignite Table</p>
  <pre><code>CREATE TABLE edw_dds_ticket (
      ..
      ..
      PRIMARY KEY (helix_uuid, ticket_issue_date)
  ) WITH &quot;TEMPLATE=PARTITIONED,backups=1,affinity_key=ticket_issue_date&quot;;
  </code></pre>
  <p>node-configuration.xml</p>
  <pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
      &lt;beans xmlns=&quot;http://www.springframework.org/schema/beans&quot;
          xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;
          xsi:schemaLocation=&quot;http://www.springframework.org/schema/beans
          http://www.springframework.org/schema/beans/spring-beans.xsd&quot;&gt;
          &lt;bean class=&quot;org.apache.ignite.configuration.IgniteConfiguration&quot;&gt;
              &lt;property name=&quot;workDirectory&quot; value=&quot;/mnt/ignite/work&quot;/&gt;
              &lt;property name=&quot;dataStorageConfiguration&quot;&gt;
                  &lt;bean class=&quot;org.apache.ignite.configuration.DataStorageConfiguration&quot;&gt;
                      &lt;property name=&quot;defaultDataRegionConfiguration&quot;&gt;
                          &lt;bean class=&quot;org.apache.ignite.configuration.DataRegionConfiguration&quot;&gt;
                              &lt;property name=&quot;checkpointPageBufferSize&quot; value=&quot;#{2048L * 1024 * 1024}&quot;/&gt;
                              &lt;property name=&quot;persistenceEnabled&quot; value=&quot;true&quot;/&gt;
                              &lt;!-- Custom region name. --&gt;
                              &lt;property name=&quot;name&quot; value=&quot;500MB_Region&quot;/&gt;
                              &lt;!-- 100 MB initial size. --&gt;
                              &lt;property name=&quot;initialSize&quot; value=&quot;#{100L * 1024 * 1024}&quot;/&gt;
                              &lt;!-- 500 MB maximum size. --&gt;
                              &lt;property name=&quot;maxSize&quot; value=&quot;#{500L * 1024 * 1024}&quot;/&gt;
                          &lt;/bean&gt;
                      &lt;/property&gt;
                      &lt;property name=&quot;writeThrottlingEnabled&quot; value=&quot;true&quot;/&gt;
                      &lt;property name=&quot;storagePath&quot; value=&quot;/mnt/ignite/data&quot;/&gt;
                      &lt;property name=&quot;walPath&quot; value=&quot;/mnt/ignite/wal&quot;/&gt;
                      &lt;!-- Disabling wal archive set same path as wal--&gt;
                      &lt;property name=&quot;walArchivePath&quot; value=&quot;/mnt/ignite/wal&quot;/&gt;
                      &lt;!--&lt;property name=&quot;walArchivePath&quot; value=&quot;/mnt/ignite/walarchive&quot;/&gt;--&gt;
                      &lt;property name=&quot;walSegmentSize&quot; value=&quot;#{256 * 1024 * 1024}&quot;/&gt;
                      &lt;property name=&quot;walCompactionEnabled&quot; value=&quot;true&quot;/&gt;
                      &lt;property name=&quot;pageSize&quot; value=&quot;#{8 * 1024}&quot;/&gt;
                  &lt;/bean&gt;
              &lt;/property&gt;
              &lt;property name=&quot;discoverySpi&quot;&gt;
                  &lt;bean class=&quot;org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi&quot;&gt;
                      &lt;property name=&quot;ipFinder&quot;&gt;
                          &lt;bean class=&quot;org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder&quot;&gt;
                              &lt;constructor-arg&gt;
                                  &lt;bean class=&quot;org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration&quot;&gt;
                                      &lt;property name=&quot;namespace&quot; value=&quot;ignite&quot; /&gt;
                                      &lt;property name=&quot;serviceName&quot; value=&quot;ignite-service&quot; /&gt;
                                  &lt;/bean&gt;
                              &lt;/constructor-arg&gt;
                          &lt;/bean&gt;
                      &lt;/property&gt;
                  &lt;/bean&gt;
              &lt;/property&gt;
          &lt;/bean&gt;
      &lt;/beans&gt;
  </code></pre>
  <p>statefulset.yaml</p>
  <pre><code># An example of a Kubernetes configuration for pod deployment.
  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    # Cluster name.
    name: ignite-cluster
    namespace: ignite
  spec:
    # The initial number of pods to be started by Kubernetes.
    replicas: 6
    serviceName: ignite
    selector:
      matchLabels:
        app: ignite
    template:
      metadata:
        labels:
          app: ignite
      spec:
        serviceAccountName: ignite
        terminationGracePeriodSeconds: 60000
        containers:
          # Custom pod name.
        - name: ignite-node
          image: apacheignite/ignite:2.13.0
          resources:
            requests:
              memory: &quot;40Gi&quot;
              cpu: &quot;1&quot;
            limits:
              memory: &quot;40Gi&quot;
              cpu: &quot;4&quot;
          env:
          - name: OPTION_LIBS
            value: ignite-kubernetes,ignite-rest-http,ignite-compress,ignite-spark-2.4,ignite-spring,ignite-indexing,ignite-log4j2,ignite-slf4j
          - name: CONFIG_URI
            value: file:///mnt/ignite/config/node-configuration.xml
          - name: JVM_OPTS
            value: &quot;-server -Xms30g -Xmx30g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:MaxDirectMemorySize=2G -DIGNITE_WAIT_FOR_BACKUPS_ON_SHUTDOWN=true -Djava.net.preferIPv4Stack=true &quot;
          - name: CONTROL_JVM_OPTS
            value: &quot;-server -Djava.net.preferIPv4Stack=true -Xms30g -Xmx30g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:MaxDirectMemorySize=2G  -DIGNITE_WAIT_FOR_BACKUPS_ON_SHUTDOWN=true&quot;
          ports:
          # Ports to open.
          - containerPort: 47100 # communication SPI port
          - containerPort: 47500 # discovery SPI port
          - containerPort: 49112 # JMX port
          - containerPort: 10800 # thin clients/JDBC driver port
          - containerPort: 8080 # REST API
          volumeMounts:
          - mountPath: /mnt/ignite/config
            name: config-vol
          - mountPath: /mnt/ignite/data
            name: data-vol
          - mountPath: /mnt/ignite/wal
            name: wal-vol
          - mountPath: /mnt/ignite/work
            name: work-vol
        securityContext:
          fsGroup: 2000 # try removing this if you have permission issues
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                - key: agentpool
                  operator: In
                  values:
                  - userpool1
        volumes:
        - name: config-vol
          configMap:
            name: ignite-configmap-with-persistence
    volumeClaimTemplates:
    - metadata:
        name: data-vol
      spec:
        accessModes: [ &quot;ReadWriteOnce&quot; ]
        storageClassName: &quot;managed-csi-premium&quot;
        resources:
          requests:
            storage: &quot;100Gi&quot; 
    - metadata:
        name: work-vol
      spec:
        accessModes: [ &quot;ReadWriteOnce&quot; ]
        storageClassName: &quot;managed-csi-premium&quot;
        resources:
          requests:
            storage: &quot;10Gi&quot; # make sure to provide enought space for your application data
    - metadata:
        name: wal-vol
      spec:
        accessModes: [ &quot;ReadWriteOnce&quot; ]
        storageClassName: &quot;managed-csi-premium&quot;
        resources:
          requests:
            storage: &quot;5Gi&quot;
  #  - metadata:
  #      name: walarchive-vol
  #    spec:
  #      accessModes: [ &quot;ReadWriteOnce&quot; ]
  #      storageClassName: &quot;managed-csi-premium&quot;
  #      resources:
  #        requests:
  #          storage: &quot;5Gi&quot;
  </code></pre>
  <p>Spark Client Connection configuration.
  spark-ignite-configmap.yaml</p>
  <pre><code>apiVersion: v1
  kind: ConfigMap
  metadata:
    name: ignite-configmap
    namespace: spark
  data:
    ignite-config.xml: |
      &lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
      &lt;beans xmlns=&quot;http://www.springframework.org/schema/beans&quot;
             xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;
             xsi:schemaLocation=&quot;http://www.springframework.org/schema/beans
              http://www.springframework.org/schema/beans/spring-beans.xsd&quot;&gt;
          &lt;!-- Imports default Ignite configuration --&gt;
          &lt;bean class=&quot;org.apache.ignite.configuration.IgniteConfiguration&quot;&gt;
              &lt;!--&lt;property name=&quot;peerClassLoadingEnabled&quot; value=&quot;true&quot;/&gt; --&gt;
              &lt;property name=&quot;clientMode&quot; value=&quot;true&quot;/&gt;
              &lt;property name=&quot;discoverySpi&quot;&gt;
                  &lt;bean class=&quot;org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi&quot;&gt;
                      &lt;property name=&quot;ipFinder&quot;&gt;
                          &lt;bean class=&quot;org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder&quot;&gt;
                              &lt;constructor-arg&gt;
                                  &lt;bean class=&quot;org.apache.ignite.kubernetes.configuration.KubernetesConnectionConfiguration&quot;&gt;
                                      &lt;property name=&quot;namespace&quot; value=&quot;ignite&quot; /&gt;
                                      &lt;property name=&quot;serviceName&quot; value=&quot;ignite-service&quot; /&gt;
                                  &lt;/bean&gt;
                              &lt;/constructor-arg&gt;
                          &lt;/bean&gt;
                      &lt;/property&gt;
                  &lt;/bean&gt;
              &lt;/property&gt;
          &lt;/bean&gt;
      &lt;/beans&gt;
  </code></pre>
  <p>spark.yaml</p>
  <pre><code>apiVersion: &quot;sparkoperator.k8s.io/v1beta2&quot;
  kind: SparkApplication
  metadata:
    name: spark-ignite
    namespace: spark
    labels:
      app: spark
  spec:
    type: Scala
    mode: cluster
    image: &quot;spark:v2.4.7_ignite&quot;
    imagePullSecrets:
      - image-pull-secret
    imagePullPolicy: Always 
    mainClass: sparkentryclass
    arguments:
      - &quot;2017-01-01&quot;
      - &quot;/ignite/config/ignite-config.xml&quot;
    mainApplicationFile: &quot;local:///opt/spark/examples/jars/IgnieDataFrame-1.0-SNAPSHOT-uber.jar&quot;
    sparkVersion: &quot;2.4.7&quot;
    volumes:
      - name: config-vol
        configMap:
          name: ignite-configmap
    restartPolicy:
      type: Never
    driver:
      cores: 1
      memory: &quot;10g&quot;
      labels:
        version: 2.4.7
      serviceAccount: spark
      volumeMounts:
        - name: config-vol
          mountPath: /ignite/config
    executor:
      cores: 3 
      instances: 5
      memory: &quot;10g&quot;
      labels:
        version: 2.4.7
      volumeMounts:
        - name: config-vol
          mountPath: /ignite/config
  </code></pre>
  <p>Spark Log: It clearly says 6 clusters(Ignite) and 6 clients(Spark Executors).
  ^-- Cluster [hosts=12, CPUs=55, servers=6, clients=6, topVer=14, minorTopVer=0]
  ^-- Network [addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.14.10], discoPort=0,</p>
  <pre><code>22/06/20 11:38:53 INFO TaskSetManager: Finished task 92.0 in stage 2.0 (TID 94) in 337817 ms on 192.168.14.14 (executor 3) (96/124)
  22/06/20 11:39:13 INFO IgniteKernal:
  Metrics for local node (to disable set 'metricsLogFrequency' to 0)
      ^-- Node [id=ec62778c, uptime=00:19:00.103]
      ^-- Cluster [hosts=12, CPUs=55, servers=6, clients=6, topVer=14, minorTopVer=0]
      ^-- Network [addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.14.10], discoPort=0, commPort=47100]
      ^-- CPU [CPUs=16, curLoad=0.07%, avgLoad=0.12%, GC=0%]
      ^-- Heap [used=359MB, free=96.05%, comm=790MB]
      ^-- Outbound messages queue [size=0]
      ^-- Public thread pool [active=0, idle=0, qSize=0]
      ^-- System thread pool [active=0, idle=1, qSize=0]
      ^-- Striped thread pool [active=0, idle=16, qSize=0]
  22/06/20 11:39:56 INFO TaskSetManager: Starting task 111.0 in stage 2.0 (TID 113, 192.168.14.12, executor 2, partition 111, PROCESS_LOCAL, 38281 bytes)
  22/06/20 11:39:56 INFO TaskSetManager: Finished task 104.0 in stage 2.0 (TID 106) in 287029 ms on 192.168.14.12 (executor 2) (97/124)
  22/06/20 11:40:01 INFO TaskSetManager: Starting task 112.0 in stage 2.0 (TID 114, 192.168.14.14, executor 3, partition 112, PROCESS_LOCAL, 35435 bytes)
  22/06/20 11:40:01 INFO TaskSetManager: Finished task 94.0 in stage 2.0 (TID 96) in 354149 ms on 192.168.14.14 (executor 3) (98/124)
  22/06/20 11:40:13 INFO IgniteKernal:
  Metrics for local node (to disable set 'metricsLogFrequency' to 0)
      ^-- Node [id=ec62778c, uptime=00:20:00.110]
      ^-- Cluster [hosts=12, CPUs=55, servers=6, clients=6, topVer=14, minorTopVer=0]
      ^-- Network [addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.14.10], discoPort=0, commPort=47100]
      ^-- CPU [CPUs=16, curLoad=0.03%, avgLoad=0.12%, GC=0%]
      ^-- Heap [used=369MB, free=95.94%, comm=790MB]
      ^-- Outbound messages queue [size=0]
      ^-- Public thread pool [active=0, idle=0, qSize=0]
      ^-- System thread pool [active=0, idle=1, qSize=0]
      ^-- Striped thread pool [active=0, idle=16, qSize=0]
  
  </code></pre>
  <p>Ignite Cluster Pods Storage Details</p>
  <pre><code>Pod-5: k exec -it ignite-cluster-5 -- du -h /mnt/ignite/data
  16.0K   /mnt/ignite/data/lost+found
  340.0K  /mnt/ignite/data/node00-3d2f2427-89f9-4950-b9ce-668864d79493/metastorage
  104.0K  /mnt/ignite/data/node00-3d2f2427-89f9-4950-b9ce-668864d79493/cache-SQL_PUBLIC_EDW_DDS_TICKET
  88.0K   /mnt/ignite/data/node00-3d2f2427-89f9-4950-b9ce-668864d79493/cp
  4.0K    /mnt/ignite/data/node00-3d2f2427-89f9-4950-b9ce-668864d79493/snp
  4.0K    /mnt/ignite/data/node00-3d2f2427-89f9-4950-b9ce-668864d79493/TxLog
  60.0K   /mnt/ignite/data/node00-3d2f2427-89f9-4950-b9ce-668864d79493/cache-ignite-sys-cache
  608.0K  /mnt/ignite/data/node00-3d2f2427-89f9-4950-b9ce-668864d79493
  632.0K  /mnt/ignite/data
   
  Pod-4: k exec -it ignite-cluster-4 -- du -h /mnt/ignite/data
  16.0K   /mnt/ignite/data/lost+found
  60.0K   /mnt/ignite/data/node00-4f28bdd5-bd15-484f-860c-fcfa665c63f1/cache-ignite-sys-cache
  4.0K    /mnt/ignite/data/node00-4f28bdd5-bd15-484f-860c-fcfa665c63f1/snp
  104.0K  /mnt/ignite/data/node00-4f28bdd5-bd15-484f-860c-fcfa665c63f1/cache-SQL_PUBLIC_EDW_DDS_TICKET
  324.0K  /mnt/ignite/data/node00-4f28bdd5-bd15-484f-860c-fcfa665c63f1/metastorage
  4.0K    /mnt/ignite/data/node00-4f28bdd5-bd15-484f-860c-fcfa665c63f1/TxLog
  72.0K   /mnt/ignite/data/node00-4f28bdd5-bd15-484f-860c-fcfa665c63f1/cp
  576.0K  /mnt/ignite/data/node00-4f28bdd5-bd15-484f-860c-fcfa665c63f1
  600.0K  /mnt/ignite/data
                                                                                                                                                                                                                                                                             Pod-3: k exec -it ignite-cluster-3 -- du -h /mnt/ignite/data
  16.0K   /mnt/ignite/data/lost+found
  24.0K   /mnt/ignite/data/node00-1ec48b28-64c0-4dde-9690-2fea32cfb1f5/cp
  316.0K  /mnt/ignite/data/node00-1ec48b28-64c0-4dde-9690-2fea32cfb1f5/metastorage
  4.0K    /mnt/ignite/data/node00-1ec48b28-64c0-4dde-9690-2fea32cfb1f5/TxLog
  18.4G   /mnt/ignite/data/node00-1ec48b28-64c0-4dde-9690-2fea32cfb1f5/cache-SQL_PUBLIC_EDW_DDS_TICKET
  60.0K   /mnt/ignite/data/node00-1ec48b28-64c0-4dde-9690-2fea32cfb1f5/cache-ignite-sys-cache
  4.0K    /mnt/ignite/data/node00-1ec48b28-64c0-4dde-9690-2fea32cfb1f5/snp
  18.4G   /mnt/ignite/data/node00-1ec48b28-64c0-4dde-9690-2fea32cfb1f5
  18.4G   /mnt/ignite/data
                                                                                                                                                                                                                                                                             Pod-2: k exec -it ignite-cluster-2 -- du -h /mnt/ignite/data
  16.0K   /mnt/ignite/data/lost+found
  4.0K    /mnt/ignite/data/node00-56ad3ba2-6d57-4405-bee9-5e155d2dffd4/snp
  308.0K  /mnt/ignite/data/node00-56ad3ba2-6d57-4405-bee9-5e155d2dffd4/metastorage
  24.0K   /mnt/ignite/data/node00-56ad3ba2-6d57-4405-bee9-5e155d2dffd4/cp
  20.7G   /mnt/ignite/data/node00-56ad3ba2-6d57-4405-bee9-5e155d2dffd4/cache-SQL_PUBLIC_EDW_DDS_TICKET
  4.0K    /mnt/ignite/data/node00-56ad3ba2-6d57-4405-bee9-5e155d2dffd4/TxLog
  60.0K   /mnt/ignite/data/node00-56ad3ba2-6d57-4405-bee9-5e155d2dffd4/cache-ignite-sys-cache
  20.7G   /mnt/ignite/data/node00-56ad3ba2-6d57-4405-bee9-5e155d2dffd4
  20.7G   /mnt/ignite/data
                                                                                                                                                                                                                                                                             Pod-1: k exec -it ignite-cluster-1 -- du -h /mnt/ignite/data
  60.0K   /mnt/ignite/data/node00-b30da2e8-4af9-492b-b15c-3371f5871508/cache-ignite-sys-cache
  4.0K    /mnt/ignite/data/node00-b30da2e8-4af9-492b-b15c-3371f5871508/TxLog
  308.0K  /mnt/ignite/data/node00-b30da2e8-4af9-492b-b15c-3371f5871508/metastorage
  4.0K    /mnt/ignite/data/node00-b30da2e8-4af9-492b-b15c-3371f5871508/snp
  28.0K   /mnt/ignite/data/node00-b30da2e8-4af9-492b-b15c-3371f5871508/cp
  2.1G    /mnt/ignite/data/node00-b30da2e8-4af9-492b-b15c-3371f5871508/cache-SQL_PUBLIC_EDW_DDS_TICKET
  2.1G    /mnt/ignite/data/node00-b30da2e8-4af9-492b-b15c-3371f5871508
  16.0K   /mnt/ignite/data/lost+found
  2.1G    /mnt/ignite/data
                                                                                                                                                                                                                                                                             Pod-1: k exec -it ignite-cluster-0 -- du -h /mnt/ignite/data
  4.0K    /mnt/ignite/data/node00-a1fbb947-6c8f-44ac-bb2c-7980f2316bb8/TxLog
  324.0K  /mnt/ignite/data/node00-a1fbb947-6c8f-44ac-bb2c-7980f2316bb8/metastorage
  60.0K   /mnt/ignite/data/node00-a1fbb947-6c8f-44ac-bb2c-7980f2316bb8/cache-ignite-sys-cache
  4.0K    /mnt/ignite/data/node00-a1fbb947-6c8f-44ac-bb2c-7980f2316bb8/snp
  104.0K  /mnt/ignite/data/node00-a1fbb947-6c8f-44ac-bb2c-7980f2316bb8/cache-SQL_PUBLIC_EDW_DDS_TICKET
  88.0K   /mnt/ignite/data/node00-a1fbb947-6c8f-44ac-bb2c-7980f2316bb8/cp
  592.0K  /mnt/ignite/data/node00-a1fbb947-6c8f-44ac-bb2c-7980f2316bb8
  16.0K   /mnt/ignite/data/lost+found
  616.0K  /mnt/ignite/data
  
  </code></pre>