<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): "panic reflect.Value.Elem on zero value", "panic", "reflect.Value.Elm"</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): Bug report</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.4+dd6b458&quot;, GitCommit:&quot;dd6b458ef8dbf24aff55795baa68f83383c9b3a9&quot;, GitTreeState:&quot;not a git tree&quot;, BuildDate:&quot;2016-08-04T09:44:57Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.4&quot;, GitCommit:&quot;dd6b458ef8dbf24aff55795baa68f83383c9b3a9&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-01T16:38:31Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">$ kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4+dd6b458", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"not a git tree", BuildDate:"2016-08-04T09:44:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:38:31Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): OS X</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ uname -a
Darwin CLI-24472 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64"><pre class="notranslate"><code class="notranslate">$ uname -a
Darwin CLI-24472 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64
</code></pre></div>
<ul dir="auto">
<li><strong>Install tools</strong>: <code class="notranslate">kubectl</code></li>
<li><strong>Others</strong>: Nothing relevant</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<ul dir="auto">
<li>Ran the following command to get the JSON path from metadata</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get rs --output=json
        {
            &quot;kind&quot;: &quot;ReplicaSet&quot;,
            &quot;apiVersion&quot;: &quot;extensions/v1beta1&quot;,
            &quot;metadata&quot;: {
                &quot;name&quot;: &quot;test-rs-12345&quot;,
                &quot;namespace&quot;: &quot;test-namespace&quot;,
                &quot;selfLink&quot;: &quot;/apis/extensions/v1beta1/namespaces/test-namespace/replicasets/test-rs-12345&quot;,
                &quot;uid&quot;: &quot;f38fdc4a-64ed-11e6-ae0f-06c010ef57b9&quot;,
                &quot;resourceVersion&quot;: &quot;2013273&quot;,
                &quot;generation&quot;: 3,
                &quot;creationTimestamp&quot;: &quot;2016-08-01T00:00:00Z&quot;,
...
      }"><pre class="notranslate"><code class="notranslate">$ kubectl get rs --output=json
        {
            "kind": "ReplicaSet",
            "apiVersion": "extensions/v1beta1",
            "metadata": {
                "name": "test-rs-12345",
                "namespace": "test-namespace",
                "selfLink": "/apis/extensions/v1beta1/namespaces/test-namespace/replicasets/test-rs-12345",
                "uid": "f38fdc4a-64ed-11e6-ae0f-06c010ef57b9",
                "resourceVersion": "2013273",
                "generation": 3,
                "creationTimestamp": "2016-08-01T00:00:00Z",
...
      }
</code></pre></div>
<ul dir="auto">
<li>Ran the following command on a namespace that has a replica set and saw the following panic message</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get rs --sort-by=.metadata.creationTimestamp
panic: reflect: call of reflect.Value.Elem on zero Value

goroutine 1 [running]:
panic(0x1552020, 0xc8200a9ea0)
        /usr/local/Cellar/go/1.6.3/libexec/src/runtime/panic.go:481 +0x3e6
reflect.Value.Elem(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.6.3/libexec/src/reflect/value.go:735 +0x228
k8s.io/kubernetes/pkg/kubectl.SortObjects(0x26f5900, 0xc8200a9dc0, 0xc8201845b0, 0xd, 0xd, 0xc8200a73e0, 0x1b, 0x0, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/sorting_printer.go:108 +0x3f1
k8s.io/kubernetes/pkg/kubectl/cmd.RunGet(0xc8204fa200, 0x26ef1c0, 0xc820030010, 0xc8200a4600, 0xc8200d63e0, 0x1, 0x2, 0xc82000ec20, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:289 +0x1758
k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdGet.func1(0xc8200a4600, 0xc8200d63e0, 0x1, 0x2)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:92 +0x7d
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc8200a4600, 0xc820235dc0, 0x2, 0x2, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:565 +0x85a
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc8200a4000, 0xc8200a4600, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:651 +0x55c
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc8200a4000, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:610 +0x2d
k8s.io/kubernetes/cmd/kubectl/app.Run(0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:32 +0xf0
main.main()
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:28 +0x28"><pre class="notranslate"><code class="notranslate">$ kubectl get rs --sort-by=.metadata.creationTimestamp
panic: reflect: call of reflect.Value.Elem on zero Value

goroutine 1 [running]:
panic(0x1552020, 0xc8200a9ea0)
        /usr/local/Cellar/go/1.6.3/libexec/src/runtime/panic.go:481 +0x3e6
reflect.Value.Elem(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.6.3/libexec/src/reflect/value.go:735 +0x228
k8s.io/kubernetes/pkg/kubectl.SortObjects(0x26f5900, 0xc8200a9dc0, 0xc8201845b0, 0xd, 0xd, 0xc8200a73e0, 0x1b, 0x0, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/sorting_printer.go:108 +0x3f1
k8s.io/kubernetes/pkg/kubectl/cmd.RunGet(0xc8204fa200, 0x26ef1c0, 0xc820030010, 0xc8200a4600, 0xc8200d63e0, 0x1, 0x2, 0xc82000ec20, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:289 +0x1758
k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdGet.func1(0xc8200a4600, 0xc8200d63e0, 0x1, 0x2)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:92 +0x7d
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc8200a4600, 0xc820235dc0, 0x2, 0x2, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:565 +0x85a
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc8200a4000, 0xc8200a4600, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:651 +0x55c
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc8200a4000, 0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:610 +0x2d
k8s.io/kubernetes/cmd/kubectl/app.Run(0x0, 0x0)
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:32 +0xf0
main.main()
        /private/tmp/kubernetes-cli-20160804-3929-1s0xod/kubernetes-1.3.4/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:28 +0x28
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
Sorted output by timestamp</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
By running this command on a NS that has a RS <code class="notranslate">$ kubectl get rs --sort-by=.metadata.creationTimestamp</code></p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">Thanks for looking into this!</p>
<p dir="auto">I am working on a Kubernetes extension that would run in its own pod but would like to store some information (e.g. configuration) that has similar requirements as the data stored in API Server.  In particular it would be nice if the storage had the following properties:</p>
<ul dir="auto">
<li><strong>Durable:</strong>  Survives pod restarts, migrations, and upgrades.</li>
<li><strong>Consistent:</strong>  Even if there are multiple pods writing the same resource concurrently, consistency is maintained.</li>
<li><strong>Addressable:</strong>  When a pod starts it can find the storage allocated for its application.</li>
<li><strong>Isolated:</strong>  Different applications don't accidentally write over each others data even if they use the same names, paths, keys, etc.</li>
<li><strong>Semi-structured:</strong>  Something like JSON-store would be fine.</li>
<li><strong>Service Oriented:</strong>  Ideally this is provided as a simple RESTful service such that it is accessible from any operating environment and doesn't require linking any specific client library.</li>
<li><strong>Timestamped:</strong>  Each resource has a logical-clock timestamp (i.e. eTag) that can be used for optimistic concurrency control.  Ideally without imposing any particular versioning scheme or strategy on the resources themselves.</li>
<li><strong>Watch Support:</strong>  It is possible to watch or be notified of updates to resources (or even small set of resources, e.g. subfolders).</li>
</ul>
<h1 dir="auto">Possibilities:</h1>
<h2 dir="auto">Use Volumes</h2>
<p dir="auto">Applications can always mount a volume and write their own storage to files.  This is not ideal for a few  reasons:</p>
<ul dir="auto">
<li>It requires a lot of reimplementation for each application.</li>
<li>Disks that survive cluster and pod restart are not a guaranteed part of all Kubernetes deployments making apps more difficult to deploy and move between Kube environments.</li>
<li>PD can have undesirable limitations (e.g. how many pods can mount them simultaneously).</li>
</ul>
<h2 dir="auto">Have applications run their own storage as a pod</h2>
<p dir="auto">Running your own etcd cluster is an obvious choice.  However, running storage as a pod is actually quite tricky.  Durability and consistency are difficult to provide and are often better maintained by a dedicated team that knows about backup and recovery.  Durability itself requires some off pod storage which brings you back to volumes.  Storage pods that are restarted need to rediscover and reattach the persistent storage used by a previous instance even though their identities are not related in any way (see Nominal Services).</p>
<h2 dir="auto">API Server's Etcd Cluster</h2>
<p dir="auto">The etcd server used by the API Server provides almost all of these features with the exception of isolation.  One possibility would be to expose a thin shim service at the API Server that wraps its underlying etcd cluster and re-exposes scoped portions of its namespace to pods.  The shim would enforce scoping and authorization.  The shim could also provide some adaption of the etcd watch interface to be more consistent with the semantics exposed by the API Server itself.</p>
<p dir="auto">Pros:  This possibility unifies the problems of durability, consistency, addressability with the API Servers almost identical requirements.  Since any deployment must already solve these issues for API Server it doesn't create any additional burden.  In hosted environments (e.g. GKE) the host provides Etcd and its durable storage in a way that is independent of the Kubernetes model allowing for backup, restore, and survivability to be implemented in ways that would be difficult if run directly as a pod.  API Server already implements a sophisticated RESTful web service endpoint with authentication, authorization, timestamping, and watch support.  This logic could be shared by the shim without addition complexity or duplication.</p>
<p dir="auto">Cons: Opening up the API Server's Etcd server (even if the shim correctly implements isolation) to third-party applications will create additional load on both API Server and Etcd.  This would affect the responsiveness of API Server and scalability in ways that may be difficult to predict.  Bugs in the shim or Etcd might expose the data from either the master or other applications to corruption or deletion.</p>