<issue_start><issue_comment>Title: WIP DO NOT MERGE: 1.3 updates
username_0: This branch will continue to be updated until there is an official 1.3 release - but want to put it up for other's to start any initial validation / testing
<issue_comment>username_1: LGTM so far as beta.2 Nice work on making the conformance tests easier for development I'll play with it soon. I'll start rebasing this into rktnetes and play with it.
<issue_comment>username_2: I launched the v15 kube-dns controller used in this deployment in an existing 1.2 cluster and the kubedns container would log each (failed?) dns request. this caused a feedback loop in the log system...

Not sure where the upstream for kube-dns is actually, so i thought i'd note it here.

Some of it surely has to do with the search path (the actual service endpoint is logs.graylog.svc.cluster.local, with fluentd being configured to use logs.graylog). The logs (millions per hour with 3 kube-dns replicas and a pretty OK logcluster) all refer to dns lookups with (broken) searchpaths, ie:
```
2016-06-29 22:49:44.000	I0629 22:49:44.024693 1 dns.go:453] records:[], retval:[], path:[local cluster svc kube-system graylog logs]
2016-06-29 22:49:44.000	I0629 22:49:44.050550 1 dns.go:453] records:[], retval:[], path:[local cluster svc kube-system graylog logs]
2016-06-29 22:49:44.000	I0629 22:49:44.054108 1 dns.go:453] records:[], retval:[], path:[local cluster svc kube-system graylog logs]
2016-06-29 22:49:44.000	I0629 22:49:44.055187 1 dns.go:421] Received DNS Request:logs.graylog.kube-system.svc.cluster.local., exact:false
```
<issue_comment>username_3: Any plans on when this will be merged to master now that 1.3 is released?
<issue_comment>username_1: @username_3 Likely by the end of this week. If you want to play with this PR in the meantime go for it! We've released a 1.3 hyperkube image at `quay.io/coreos/hyperkube:v1.3.0_coreos.0`.
<issue_comment>username_0: Last conformance run failures:

```
Summarizing 21 Failures:

[Fail] [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking [It] resource tracking for 100 pods per node 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:104

[Fail] [k8s.io] KubeletManagedEtcHosts [It] should test kubelet managed /etc/hosts file 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_etc_hosts.go:139

[Fail] [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:284

[Fail] [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:284

[Fail] [k8s.io] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow] 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2614

[Fail] [k8s.io] Networking [k8s.io] Granular Checks [It] should function for pod communication on a single node 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:4316

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] [Serial] [Slow] ReplicaSet [It] Should scale from 1 pod to 3 pods and from 3 to 5 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:284

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that a pod with an invalid podAffinity is rejected because of the LabelSelectorRequirement is invalid 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that required NodeAffinity setting is respected if matching 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] Daemon set [Serial] [It] should run and stop complex daemon 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:194

[Fail] [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking [It] resource tracking for 35 pods per node 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:77

[Fail] [k8s.io] KubeProxy [It] should test kube-proxy [Slow] 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubeproxy.go:521

[Fail] [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking [It] resource tracking for 0 pods per node 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:104

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that taints-tolerations is respected if not matching 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates resource limits of pods that are allowed to run [Conformance] 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that a pod with an invalid NodeAffinity is rejected 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] [Serial] [Slow] Deployment [It] Should scale from 5 pods to 3 pods and from 3 to 1 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:351

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that NodeSelector is respected if matching [Conformance] 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] NodeProblemDetector [k8s.io] KernelMonitor [BeforeEach] should generate node condition and events for corresponding errors 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node_problem_detector.go:139

[Fail] [k8s.io] Services [It] should be able to create a functioning NodePort service 
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1732

Ran 203 of 324 Specs in 35725.460 seconds
FAIL! -- 182 Passed | 21 Failed | 0 Pending | 121 Skipped --- FAIL: TestE2E (35725.47s)
```
<issue_comment>username_0: I ran all non-serial tests 3 times, and any of those errors seem to be flakes (two runs had 1 error each -- different each time, and one run had zero errors).

That leaves us with 16 outstanding errors:

- 8 are related to autoscaler / resource tracking (likely candidates are cadvisor / cgroup incompatibilities and/or heapster misconfiguration).

```
[Fail] [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking [It] resource tracking for 100 pods per node
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:104

[Fail] [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] [Serial] [Slow] ReplicationController [It] Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:284

[Fail] [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] [Serial] [Slow] ReplicationController [It] Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:284

[Fail] [k8s.io] SchedulerPredicates [Serial] [It] validates MaxPods limit number of pods that are allowed to run [Slow]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2614

[Fail] [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] [Serial] [Slow] ReplicaSet [It] Should scale from 1 pod to 3 pods and from 3 to 5
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:284

[Fail] [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking [It] resource tracking for 35 pods per node
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:77

[Fail] [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking [It] resource tracking for 0 pods per node
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:104

[Fail] [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] [Serial] [Slow] Deployment [It] Should scale from 5 pods to 3 pods and from 3 to 1
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:351

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates resource limits of pods that are allowed to run [Conformance]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211
```

* 6 of the errors are related to scheduler / scheduler_predicates

```
[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that a pod with an invalid podAffinity is rejected because of the LabelSelectorRequirement is invalid
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that required NodeAffinity setting is respected if matching
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that taints-tolerations is respected if not matching
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that a pod with an invalid NodeAffinity is rejected
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

[Fail] [k8s.io] SchedulerPredicates [Serial] [BeforeEach] validates that NodeSelector is respected if matching [Conformance]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211
```

And 1 misc error: 

```
[Fail] [k8s.io] Daemon set [Serial] [It] should run and stop complex daemon
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:194
```
<issue_comment>username_0: Re-ran conformance tests using `--ginkgo.skip=\[Flaky\]|\[Feature:.+\]`

Still have one consistent failure: `validates resource limits of pods that are allowed to run`

Test failure:

```
• Failure [980.440 seconds]
[k8s.io] SchedulerPredicates [Serial]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:685
  validates resource limits of pods that are allowed to run [Conformance] [It]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:389

  Error waiting for 6 pods to be running - probably a timeout
  Expected error:
      <*errors.errorString | 0xc821afa140>: {
          s: "Timeout while waiting for pods with labels \"startPodsID=61b2350d-455a-11e6-bc42-0242ac110002\" to be running",
      }
      Timeout while waiting for pods with labels "startPodsID=61b2350d-455a-11e6-bc42-0242ac110002" to be running
  not to have occurred

  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2614
```
<issue_comment>username_0: @username_4 @username_1 

Any thoughts above moving forward with merging this? If so, I will open a separate issue to try and track down the root cause for that last failure.
<issue_comment>username_4: @username_0 if that's the only consistent failure, I'm all for it.
<issue_comment>username_4: We can wait till the error is sorted before we cut the next kube-aws release.