<p dir="auto">Go version: go1.3.3 linux/amd64 on Fedora 20</p>
<p dir="auto">When creating an archive (with <code class="notranslate">tar.NewWriter</code>), long names (&gt; 100 chars) that don't fit into standard tar headers need to be encoded differently.<br>
An optimisation in <code class="notranslate">archive/tar/writer.go: writeHeader()</code> tries to use a ustar header when only the name is too long creates files that are misinterpretated by other tar implementations (but read correctly by archive/tar)</p>
<p dir="auto">For example, <code class="notranslate">/home/support/.openoffice.org/3/user/uno_packages/cache/registry/com.sun.star.comp.deployment.executable.PackageRegistryBackend</code> becomes <code class="notranslate">com.sun.star.comp.deployment.executable.PackageRegistryBackend</code> for external <code class="notranslate">tar</code> commands (tested with GNU tar, BSD tar and star)</p>
<p dir="auto">Modifying <code class="notranslate">archive/tar/writer.go</code> and forcing <code class="notranslate">preferPax</code> to <code class="notranslate">true</code> in <code class="notranslate">NewWriter</code> fixes the issue</p>
<p dir="auto">Using <code class="notranslate">go1.5</code></p>
<p dir="auto">Also discovered this while fixing other archive/tar issues (and I found fair number of them, mostly minor). However, fixing this will change the way archive/tar reads and writes certain formats.</p>
<h4 dir="auto">What the current archive/tar thinks the GNU format is:</h4>
<ul dir="auto">
<li>A magic and version that forms the string <a href="https://github.com/golang/go/blob/a1d093d95dfb888e55f960943bd3ad3d181ec743/src/archive/tar/reader.go#L493"><code class="notranslate">"ustar\x20\x20\x00"</code></a> (this is correct).</li>
<li>That the structure is <a href="https://github.com/golang/go/blob/a1d093d95dfb888e55f960943bd3ad3d181ec743/src/archive/tar/reader.go#L510">identical to the POSIX format</a>. That is, there is a 155byte prefix section (this is incorrect).</li>
<li>That it extends the POSIX format by <a href="https://github.com/golang/go/blob/a1d093d95dfb888e55f960943bd3ad3d181ec743/src/archive/tar/writer.go#L211">adding the ability to perform base-256 encoding</a> (this is not necessarily specific to GNU format).</li>
</ul>
<h4 dir="auto">What the GNU manual actually says the format is:</h4>
<p dir="auto">The <a href="https://www.gnu.org/software/tar/manual/html_node/Standard.html" rel="nofollow">GNU manual</a> says that the format for headers using this magic is the following (in Go syntax):</p>
<div class="highlight highlight-source-go notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="type headerGNU struct {
    // Original V7 header
    name     [100]byte //   0
    mode     [8]byte   // 100
    uid      [8]byte   // 108
    gid      [8]byte   // 116
    size     [12]byte  // 124
    mtime    [12]byte  // 136
    chksum   [8]byte   // 148
    typeflag [1]byte   // 156
    linkname [100]byte // 157

    // This section is based on the Posix standard.
    magic      [6]byte         // 257: &quot;ustar &quot;
    version    [2]byte         // 263: &quot; \x00&quot;
    uname      [32]byte        // 265
    gname      [32]byte        // 297
    devmajor   [8]byte         // 329
    devminor   [8]byte         // 337

    // The GNU format replaces the prefix field with this stuff.
    // The fact that GNU replaces the prefix with this makes it non-compliant.
    atime      [12]byte        // 345
    ctime      [12]byte        // 357
    offset     [12]byte        // 369
    longnames  [4]byte         // 381
    unused     [1]byte         // 385
    sparse     [4]headerSparse // 386
    isextended [1]byte         // 482
    realsize   [12]byte        // 483
                               // 495
}

type headerSparse struct {
    offset   [12]byte //  0
    numbytes [12]byte // 12
                      // 24
}"><pre class="notranslate"><span class="pl-k">type</span> <span class="pl-smi">headerGNU</span> <span class="pl-k">struct</span> {
    <span class="pl-c">// Original V7 header</span>
    <span class="pl-c1">name</span>     [<span class="pl-c1">100</span>]<span class="pl-smi">byte</span> <span class="pl-c">//   0</span>
    <span class="pl-c1">mode</span>     [<span class="pl-c1">8</span>]<span class="pl-smi">byte</span>   <span class="pl-c">// 100</span>
    <span class="pl-c1">uid</span>      [<span class="pl-c1">8</span>]<span class="pl-smi">byte</span>   <span class="pl-c">// 108</span>
    <span class="pl-c1">gid</span>      [<span class="pl-c1">8</span>]<span class="pl-smi">byte</span>   <span class="pl-c">// 116</span>
    <span class="pl-c1">size</span>     [<span class="pl-c1">12</span>]<span class="pl-smi">byte</span>  <span class="pl-c">// 124</span>
    <span class="pl-c1">mtime</span>    [<span class="pl-c1">12</span>]<span class="pl-smi">byte</span>  <span class="pl-c">// 136</span>
    <span class="pl-c1">chksum</span>   [<span class="pl-c1">8</span>]<span class="pl-smi">byte</span>   <span class="pl-c">// 148</span>
    <span class="pl-c1">typeflag</span> [<span class="pl-c1">1</span>]<span class="pl-smi">byte</span>   <span class="pl-c">// 156</span>
    <span class="pl-c1">linkname</span> [<span class="pl-c1">100</span>]<span class="pl-smi">byte</span> <span class="pl-c">// 157</span>

    <span class="pl-c">// This section is based on the Posix standard.</span>
    <span class="pl-c1">magic</span>      [<span class="pl-c1">6</span>]<span class="pl-smi">byte</span>         <span class="pl-c">// 257: "ustar "</span>
    <span class="pl-c1">version</span>    [<span class="pl-c1">2</span>]<span class="pl-smi">byte</span>         <span class="pl-c">// 263: " \x00"</span>
    <span class="pl-c1">uname</span>      [<span class="pl-c1">32</span>]<span class="pl-smi">byte</span>        <span class="pl-c">// 265</span>
    <span class="pl-c1">gname</span>      [<span class="pl-c1">32</span>]<span class="pl-smi">byte</span>        <span class="pl-c">// 297</span>
    <span class="pl-c1">devmajor</span>   [<span class="pl-c1">8</span>]<span class="pl-smi">byte</span>         <span class="pl-c">// 329</span>
    <span class="pl-c1">devminor</span>   [<span class="pl-c1">8</span>]<span class="pl-smi">byte</span>         <span class="pl-c">// 337</span>

    <span class="pl-c">// The GNU format replaces the prefix field with this stuff.</span>
    <span class="pl-c">// The fact that GNU replaces the prefix with this makes it non-compliant.</span>
    <span class="pl-c1">atime</span>      [<span class="pl-c1">12</span>]<span class="pl-smi">byte</span>        <span class="pl-c">// 345</span>
    <span class="pl-c1">ctime</span>      [<span class="pl-c1">12</span>]<span class="pl-smi">byte</span>        <span class="pl-c">// 357</span>
    <span class="pl-c1">offset</span>     [<span class="pl-c1">12</span>]<span class="pl-smi">byte</span>        <span class="pl-c">// 369</span>
    <span class="pl-c1">longnames</span>  [<span class="pl-c1">4</span>]<span class="pl-smi">byte</span>         <span class="pl-c">// 381</span>
    <span class="pl-c1">unused</span>     [<span class="pl-c1">1</span>]<span class="pl-smi">byte</span>         <span class="pl-c">// 385</span>
    <span class="pl-c1">sparse</span>     [<span class="pl-c1">4</span>]<span class="pl-smi">headerSparse</span> <span class="pl-c">// 386</span>
    <span class="pl-c1">isextended</span> [<span class="pl-c1">1</span>]<span class="pl-smi">byte</span>         <span class="pl-c">// 482</span>
    <span class="pl-c1">realsize</span>   [<span class="pl-c1">12</span>]<span class="pl-smi">byte</span>        <span class="pl-c">// 483</span>
                               <span class="pl-c">// 495</span>
}

<span class="pl-k">type</span> <span class="pl-smi">headerSparse</span> <span class="pl-k">struct</span> {
    <span class="pl-c1">offset</span>   [<span class="pl-c1">12</span>]<span class="pl-smi">byte</span> <span class="pl-c">//  0</span>
    <span class="pl-c1">numbytes</span> [<span class="pl-c1">12</span>]<span class="pl-smi">byte</span> <span class="pl-c">// 12</span>
                      <span class="pl-c">// 24</span>
}</pre></div>
<p dir="auto">In fact, the structure for GNU swaps out the prefix section of POSIX, for a bunch of extra fields for atime, ctime, and sparse file support (contrary to what Go thinks).</p>
<p dir="auto">Regarding the use of base-256 encoding, it seems that GNU was the first to <a href="http://git.savannah.gnu.org/cgit/tar.git/commit/?id=e4e624848b53ac02f1212af2209a63d28e40afec" rel="nofollow">introduce this encoding  back in 1999</a>. Since then, pretty much every tar decoder handles reading base-256 encoding regardless of whether it is GNU format or not. Marking the format as GNU may or may not be necessary just because base-256 encoding was used.</p>
<h4 dir="auto">Problem 1:</h4>
<p dir="auto">When reading, if the decoder detects the GNU magic number, it will attempt to read 155bytes for the prefix. This is just plain wrong and will start to read the atime, ctime, etc instead. This causes the prefix to be incorrect.</p>
<p dir="auto"><a href="https://play.golang.org/p/oARo0DMzys" rel="nofollow">See this playground example</a><br>
The paths there have something like "12574544345" prepended to it. This is because when the tar archive tries to read the the prefix, it is actually reading the atime (which is in ASCII octal and is null terminated). Thus, it incorrectly uses the atime as the prefix.</p>
<p dir="auto">This probably went undetected for so long since the "incremental" mode of GNU tar is rarely used, and thus the atime and ctime fields are never filled out and left as null bytes. This happens to work in the common case, since the cstring for this field ends up being an empty string.</p>
<h4 dir="auto">Problem 2:</h4>
<p dir="auto">When writing, if a numeric field was ever too large to represent in octal format, it would trigger the <code class="notranslate">usedBinary</code> flag and cause the library to output the GNU magic numbers, but subsequently fail to encode in the GNU format. Since it believes that the GNU format has a prefix field, it erroneously tries to use it, losing some information in the process.</p>
<p dir="auto">This is ultimately what causes <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="55409886" data-permission-text="Title is private" data-url="https://github.com/golang/go/issues/9683" data-hovercard-type="issue" data-hovercard-url="/golang/go/issues/9683/hovercard" href="https://github.com/golang/go/issues/9683">#9683</a>, but is rare in practice since the perfect conditions need to be met for GNU format to be used. There is a very narrow range between the use cases of USTAR and PAX where the logic will use GNU.</p>
<h4 dir="auto">Solution:</h4>
<p dir="auto">When decoding, change it so that the reader doesn't read the 155byte prefix field (since this is just plain wrong). Optionally, support parsing of the atime and ctime from the GNU format. Nothing needs to change for sparse file support since that logic correctly understood the GNU format.</p>
<p dir="auto">When encoding, I propose the following order of precedence:</p>
<ul dir="auto">
<li>First, use the 1988 POSIX (USTAR) standard when possible for maximum backwards compatibility.</li>
<li>If any numeric field goes beyond the octal representation, or any names are longer than what is supported, just use the 2001 POSIX (PAX) standard.</li>
</ul>
<p dir="auto">Let's avoid writing the GNU format. In fact the GNU manual itself, <a href="https://www.gnu.org/software/tar/manual/html_node/Formats.html#SEC132" rel="nofollow">says the following under the POSIX section</a>:</p>
<blockquote>
<p dir="auto">This archive format will be the default format for future versions of GNU tar.</p>
</blockquote>
<p dir="auto">The only advantages that GNU offers over USTAR is:</p>
<ul dir="auto">
<li>Unlimited length filenames (only ASCII)</li>
<li>Relatively large filesizes</li>
<li>Possibly atime and ctime</li>
</ul>
<p dir="auto">However, PAX offers all of these over USTAR and far more:</p>
<ul dir="auto">
<li>Unlimited length strings (including UTF-8) support for filenames, usernames, etc.</li>
<li>Unlimited large integers for filesizes, uids, etc.</li>
<li>Sub-second resolution times.</li>
<li>No need for base-256 encoding (and assuming that decoders can handle them) since PAX has its own well-defined method of encoding arbitrarily large integers.</li>
</ul>
<p dir="auto">Not to mention, we are already outputting PAX in many situations. What's the point of straggling between 3 different output formats?</p>
<p dir="auto">Thoughts?</p>