title: Copying a 1TB sparse file
body: <p>I got a sparse file of 1TB which stores actually 32MB data on Linux.</p>
  
  <p>Is it possible to "efficiently" make a package to store the sparse file? The package should be unpacked to be a 1TB sparse file on another computer. Ideally, the "package" should be around 32MB.</p>
  
  <p>Note: On possible solution is to use 'tar': <a href="https://wiki.archlinux.org/index.php/Sparse_file#Archiving_with_.60tar.27" rel="noreferrer">https://wiki.archlinux.org/index.php/Sparse_file#Archiving_with_.60tar.27</a></p>
  
  <p>However, for a 1TB sparse file, although the tar ball may be small, archiving the sparse file will take too long a time.</p>
  
  <p><strong>Edit 1</strong></p>
  
  <p>I tested the tar and gzip and the results are as follows (Note that this sparse file contains data of 0 byte).</p>
  
  <pre><code>$ du -hs sparse-1
  0   sparse-1
  
  $ ls -lha sparse-1
  -rw-rw-r-- 1 user1 user1 1.0T 2012-11-03 11:17 sparse-1
  
  $ time tar cSf sparse-1.tar sparse-1
  
  real    96m19.847s
  user    22m3.314s
  sys     52m32.272s
  
  $ time gzip sparse-1
  
  real    200m18.714s
  user    164m33.835s
  sys     10m39.971s
  
  $ ls -lha sparse-1*
  -rw-rw-r-- 1 user1 user1 1018M 2012-11-03 11:17 sparse-1.gz
  -rw-rw-r-- 1 user1 user1   10K 2012-11-06 23:13 sparse-1.tar
  </code></pre>
  
  <p>The 1TB file sparse-1 which contains 0 byte data can be archived by 'tar' to a 10KB tar ball or compressed by gzip to a ~1GB file. gzip takes around 2 times of the time than the time tar uses.</p>
  
  <p>From the comparison, 'tar' seems better than gzip.</p>
  
  <p>However, 96 minutes are too long for a sparse file that contains data of 0 byte.</p>
  
  <p><strong>Edit 2</strong></p>
  
  <p><code>rsync</code> seems finish copying the file in more time than <code>tar</code> but less than <code>gzip</code>:</p>
  
  <pre><code>$ time rsync --sparse sparse-1 sparse-1-copy
  
  real    124m46.321s
  user    107m15.084s
  sys     83m8.323s
  
  $ du -hs sparse-1-copy 
  4.0K    sparse-1-copy
  </code></pre>
  
  <p>Hence, <code>tar</code> + <code>cp</code> or <code>scp</code> should be faster than directly <code>rsync</code> for this extremely sparse file.</p>
  
  <p><strong>Edit 3</strong></p>
  
  <p>Thanks to @mvp for pointing out the SEEK_HOLE functionality in newer kernel. (I previously work on a 2.6.32 Linux kernel).</p>
  
  <p>Note: bsdtar version >=3.0.4 is required (check here: <a href="http://ask.fclose.com/4/how-to-efficiently-archive-a-very-large-sparse-file?show=299#c299" rel="noreferrer">http://ask.fclose.com/4/how-to-efficiently-archive-a-very-large-sparse-file?show=299#c299</a> ).</p>
  
  <p>On a newer kernel and Fedora release (17), <code>tar</code> and <code>cp</code> handles the sparse file <strong>very</strong> efficiently.</p>
  
  <pre><code>[zma@office tmp]$ ls -lh pmem-1 
  
  -rw-rw-r-- 1 zma zma 1.0T Nov  7 20:14 pmem-1
  [zma@office tmp]$ time tar cSf pmem-1.tar pmem-1
  
  real    0m0.003s
  user    0m0.003s
  sys 0m0.000s
  [zma@office tmp]$ time cp pmem-1 pmem-1-copy
  
  real    0m0.020s
  user    0m0.000s
  sys 0m0.003s
  [zma@office tmp]$ ls -lh pmem*
  -rw-rw-r-- 1 zma zma 1.0T Nov  7 20:14 pmem-1
  -rw-rw-r-- 1 zma zma 1.0T Nov  7 20:15 pmem-1-copy
  -rw-rw-r-- 1 zma zma  10K Nov  7 20:15 pmem-1.tar
  [zma@office tmp]$ mkdir t
  [zma@office tmp]$ cd t
  [zma@office t]$ time tar xSf ../pmem-1.tar 
  
  real    0m0.003s
  user    0m0.000s
  sys 0m0.002s
  [zma@office t]$ ls -lha
  total 8.0K
  drwxrwxr-x   2 zma  zma  4.0K Nov  7 20:16 .
  drwxrwxrwt. 35 root root 4.0K Nov  7 20:16 ..
  -rw-rw-r--   1 zma  zma  1.0T Nov  7 20:14 pmem-1
  </code></pre>
  
  <p>I am using a 3.6.5 kernel:</p>
  
  <pre><code>[zma@office t]$ uname -a
  Linux office.zhiqiangma.com 3.6.5-1.fc17.x86_64 #1 SMP Wed Oct 31 19:37:18 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
  </code></pre>