Issue title: roachtest: multitenant-upgrade failed
 Issue body: roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/viewLog.html?buildId=3411578&tab=buildLog) with [artifacts](https://teamcity.cockroachdb.com/viewLog.html?buildId=3411578&tab=artifacts#/multitenant-upgrade) on master @ [642e44afbe6098f25022618be76c6f7b6b97df45](https://github.com/cockroachdb/cockroach/commits/642e44afbe6098f25022618be76c6f7b6b97df45):


```
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1991
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1970
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1891
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1893
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1893
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatalf
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:264
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:59
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:222
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:323
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) cluster.RunL
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
```
<details><summary>Reproduce</summary>
<p>


See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

</p>
</details>
/cc @cockroachdb/kv-triage
<sub>

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)
</sub>

 Comments: 
 Comment 0: ```
11:18:19 multitenant_upgrade.go:309: test status: migrating the tenant 11 to the current version before kv is finalized
11:23:23 multitenant_upgrade.go:321: test status: finalizing the kv server
11:24:08 test_impl.go:323: test failure: 	soon.go:59,soon.go:41,sql_runner.go:222,multitenant_upgrade.go:323,multitenant_upgrade.go:39,test_runner.go:777: condition failed to evaluate within 45s: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
		true
		got:
		false
```

In tenant 11's logs:

```
I210908 11:18:+1-205-728-7199 migration/migrationmanager/manager.go:115 â‹® [sql,client=â€¹116.69.203.115:44454â€º,hostnossl,user=root,migration-mgr] 35  migrating cluster from 21.1-124 to 21.1-1160 (stepping through [21.1-1102 21.1-1104 21.1-1106 21.1-1108 21.1-1110 21.1-1112 21.1-1114 21.1-1116 21.1-1118 21.1-1120 21.1-1122 21.1-1124 21.1-1126 21.1-1128 21.1-1130 21.1-1132 21.1-1134 21.1-1136 21.1-1138 21.1-1140 21.1-1142 21.1-1144 21.1-1146 21.1-1148 21.1-1150 21.1-1152 21.1-1154 21.1-1156 21.1-1158 21.1-1160])
[...]
I210908 11:23:23.030275 137 server/settingswatcher/settings_watcher.go:112 â‹® [rangefeed=â€¹settingsâ€º] 321  set cluster version to: 21.1-1160
```

so it took like five minutes to migrate, much in excess of the 45s the test restricts it to. I think the test is just using a vanilla SucceedsSoon, so of course it wouldn't be appropriate now that we have actual long running migrations in that path. Nevertheless, it seems to mostly pass, so something was extraordinarily slow this time around. The vast majority of time (think all of it) is spent here:

```
I210908 11:18:19.922184 137 server/settingswatcher/settings_watcher.go:112 â‹® [rangefeed=â€¹settingsâ€º] 57  set cluster version to: 21.1-1112
I210908 11:23:21.483449 137 server/settingswatcher/settings_watcher.go:112 â‹® [rangefeed=â€¹settingsâ€º] 135  set cluster version to: 21.1-1114
```

1114 is the fix_descriptor_migration.go migration. Here's tenant 11's log grepped for 1114:

```
I210908 11:18:+1-205-728-7199 migration/migrationmanager/manager.go:115 â‹® [sql,client=â€¹116.69.203.115:44454â€º,hostnossl,user=root,migration-mgr] 35  migrating cluster from 21.1-124 to 21.1-1160 (stepping through [21.1-1102 21.1-1104 21.1-1106 21.1-1108 21.1-1110 21.1-1112 21.1-1114 21.1-1116 21.1-1118 21.1-1120 21.1-1122 21.1-1124 21.1-1126 21.1-1128 21.1-1130 21.1-1132 21.1-1134 21.1-1136 21.1-1138 21.1-1140 21.1-1142 21.1-1144 21.1-1146 21.1-1148 21.1-1150 21.1-1152 21.1-1154 21.1-1156 21.1-1158 21.1-1160])
I210908 11:18:19.920763 214 migration/migrationmanager/manager.go:135 â‹® [sql,client=â€¹116.69.203.115:44454â€º,hostnossl,user=root,migration-mgr] 56  stepping through 21.1-1114
I210908 11:18:19.984511 309 migration/migrations/fix_descriptor_migration.go:108 â‹® [sql,job=â€¹691501465340608513â€º,migration=21.1-1114] 61  upgrading descriptor with ids [1 3 4 6 7 8 11 12 13 14 15 19 20 21 23 24 25 26 27 28 31 32 33 34 35 36 37 39 40 50 51 52]
I210908 11:18:30.168223 309 sql/catalog/lease/lease.go:139 â‹® [sql,job=â€¹691501465340608513â€º,migration=21.1-1114] 64  waiting for 1 leases to expire: desc=[{â€¹systemâ€º 1 1}]
I210908 11:18:30.169291 328 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 65  new lease: tabledesc.immutable: {ID: 6, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 5, Columns: [{ID: 1, TypeID: 25, Null: false}, {ID: 2, TypeID: 25, Null: false}, {ID: 3, TypeID: 1114, Null: false, HasDefault: true}, {ID: 4, TypeID: 25, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4]}], PrimaryIndex: 1, NextIndexID: 2, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4]}]}
I210908 11:18:30.176490 329 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 66  new lease: tabledesc.immutable: {ID: 15, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 10, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 25, Null: false}, {ID: 3, TypeID: 1114, Null: false, HasDefault: true}, {ID: 4, TypeID: 17, Null: false}, {ID: 5, TypeID: 17, Null: true}, {ID: 6, TypeID: 25, Null: true}, {ID: 7, TypeID: 20, Null: true}, {ID: 8, TypeID: 17, Null: true}, {ID: 9, TypeID: 20, Null: true}], NextFamilyID: 3, Families: [{ID: 0, Columns: [1, 2, 3, 4, 6, 7]}, {ID: 1, Columns: [5]}, {ID: 2, Columns: [8, 9]}], PrimaryIndex: 1, NextIndexID: 4, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7, 8, 9]}, {ID: 2, Unique: false, KeyColumns: [{ID: 2, Dir: ASC}, {ID: 3, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 3, Unique: false, KeyColumns: [{ID: 6, Dir: ASC}, {ID: 7, Dir: ASC}], KeySuffixColumns: [1], StoreColumns: [2]}]}
I210908 11:18:30.182671 361 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 67  new lease: tabledesc.immutable: {ID: 20, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 10, Columns: [{ID: 1, TypeID: 20, Null: false}, {ID: 2, TypeID: 20, Null: false, HasDefault: true}, {ID: 3, TypeID: 25, Null: true}, {ID: 4, TypeID: 1016, Null: false}, {ID: 5, TypeID: 1114, Null: false, HasDefault: true}, {ID: 6, TypeID: 20, Null: false}, {ID: 7, TypeID: 20, Null: false}, {ID: 8, TypeID: 20, Null: false}, {ID: 9, TypeID: 17, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7, 8, 9]}], PrimaryIndex: 1, NextIndexID: 2, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}, {ID: 2, Dir: ASC}], StoreColumns: [3, 4, 5, 6, 7, 8, 9]}]}
I210908 11:23:21.483449 137 server/settingswatcher/settings_watcher.go:112 â‹® [rangefeed=â€¹settingsâ€º] 135  set cluster version to: 21.1-1114
I210908 11:23:21.+1-205-728-7199 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 187  new lease: tabledesc.immutable: {ID: 19, Version: 5, ModificationTime: "1631100201.745689121,0", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 9, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 17, Null: false}, {ID: 3, TypeID: 25, Null: false}, {ID: 4, TypeID: 1114, Null: false, HasDefault: true}, {ID: 5, TypeID: 1114, Null: false}, {ID: 6, TypeID: 1114, Null: true}, {ID: 7, TypeID: 1114, Null: false, HasDefault: true}, {ID: 8, TypeID: 25, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7, 8]}], PrimaryIndex: 1, NextIndexID: 5, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7, 8]}, {ID: 2, Unique: false, KeyColumns: [{ID: 5, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 3, Unique: false, KeyColumns: [{ID: 4, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 4, Unique: false, KeyColumns: [{ID: 6, Dir: ASC}], KeySuffixColumns: [1]}]}
I210908 11:23:21.871347 1390 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 199  new lease: tabledesc.immutable: {ID: 19, Version: 6, ModificationTime: "1631100201.851597675,0", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 9, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 17, Null: false}, {ID: 3, TypeID: 25, Null: false}, {ID: 4, TypeID: 1114, Null: false, HasDefault: true}, {ID: 5, TypeID: 1114, Null: false}, {ID: 6, TypeID: 1114, Null: true}, {ID: 7, TypeID: 1114, Null: false, HasDefault: true}, {ID: 8, TypeID: 25, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7, 8]}], MutationJobs: [{MutationID: 2, JobID: 691502454713876481}], Mutations: [{MutationID: 2, Direction: ADD, State: DELETE_ONLY, Index: {ID: 5, Unique: false, KeyColumns: [{ID: 7, Dir: ASC}], KeySuffixColumns: [1], State: ADD, MutationID: 2}}], PrimaryIndex: 1, NextIndexID: 6, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7, 8]}, {ID: 2, Unique: false, KeyColumns: [{ID: 5, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 3, Unique: false, KeyColumns: [{ID: 4, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 4, Unique: false, KeyColumns: [{ID: 6, Dir: ASC}], KeySuffixColumns: [1]}]}
I210908 11:23:21.+1-205-728-7199 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 202  new lease: tabledesc.immutable: {ID: 19, Version: 7, ModificationTime: "1631100201.884358307,0", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 9, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 17, Null: false}, {ID: 3, TypeID: 25, Null: false}, {ID: 4, TypeID: 1114, Null: false, HasDefault: true}, {ID: 5, TypeID: 1114, Null: false}, {ID: 6, TypeID: 1114, Null: true}, {ID: 7, TypeID: 1114, Null: false, HasDefault: true}, {ID: 8, TypeID: 25, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7, 8]}], MutationJobs: [{MutationID: 2, JobID: 691502454713876481}], Mutations: [{MutationID: 2, Direction: ADD, State: DELETE_AND_WRITE_ONLY, Index: {ID: 5, Unique: false, KeyColumns: [{ID: 7, Dir: ASC}], KeySuffixColumns: [1], State: ADD, MutationID: 2}}], PrimaryIndex: 1, NextIndexID: 6, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7, 8]}, {ID: 2, Unique: false, KeyColumns: [{ID: 5, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 3, Unique: false, KeyColumns: [{ID: 4, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 4, Unique: false, KeyColumns: [{ID: 6, Dir: ASC}], KeySuffixColumns: [1]}]}
I210908 11:23:22.018249 1456 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 218  new lease: tabledesc.immutable: {ID: 19, Version: 8, ModificationTime: "1631100202.001709697,0", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 9, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 17, Null: false}, {ID: 3, TypeID: 25, Null: false}, {ID: 4, TypeID: 1114, Null: false, HasDefault: true}, {ID: 5, TypeID: 1114, Null: false}, {ID: 6, TypeID: 1114, Null: true}, {ID: 7, TypeID: 1114, Null: false, HasDefault: true}, {ID: 8, TypeID: 25, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7, 8]}], PrimaryIndex: 1, NextIndexID: 6, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7, 8]}, {ID: 2, Unique: false, KeyColumns: [{ID: 5, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 3, Unique: false, KeyColumns: [{ID: 4, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 4, Unique: false, KeyColumns: [{ID: 6, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 5, Unique: false, KeyColumns: [{ID: 7, Dir: ASC}], KeySuffixColumns: [1]}]}
I210908 11:23:22.+1-205-728-7199 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 261  new lease: tabledesc.immutable: {ID: 15, Version: 7, ModificationTime: "1631100202.595460880,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 12, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 25, Null: false}, {ID: 3, TypeID: 1114, Null: false, HasDefault: true}, {ID: 4, TypeID: 17, Null: false}, {ID: 5, TypeID: 17, Null: true}, {ID: 6, TypeID: 25, Null: true}, {ID: 7, TypeID: 20, Null: true}, {ID: 8, TypeID: 17, Null: true}, {ID: 9, TypeID: 20, Null: true}, {ID: 10, TypeID: 20, Null: true}, {ID: 11, TypeID: 1114, Null: true}], NextFamilyID: 3, Families: [{ID: 0, Columns: [1, 2, 3, 4, 6, 7]}, {ID: 1, Columns: [5]}, {ID: 2, Columns: [8, 9, 10, 11]}], MutationJobs: [{MutationID: 2, JobID: 691502457005375489}], Mutations: [{MutationID: 2, Direction: ADD, State: DELETE_AND_WRITE_ONLY, Index: {ID: 4, Unique: false, Partial: true, KeyColumns: [{ID: 8, Dir: ASC}, {ID: 2, Dir: ASC}, {ID: 3, Dir: ASC}], KeySuffixColumns: [1], StoreColumns: [11, 10, 9], State: ADD, MutationID: 2}}], PrimaryIndex: 1, NextIndexID: 5, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}, {ID: 2, Unique: false, KeyColumns: [{ID: 2, Dir: ASC}, {ID: 3, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 3, Unique: false, KeyColumns: [{ID: 6, Dir: ASC}, {ID: 7, Dir: ASC}], KeySuffixColumns: [1], StoreColumns: [2]}]}
I210908 11:23:22.745817 1708 sql/catalog/lease/descriptor_state.go:139 â‹® [-] 275  new lease: tabledesc.immutable: {ID: 15, Version: 8, ModificationTime: "1631100202.728612501,0", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 12, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 25, Null: false}, {ID: 3, TypeID: 1114, Null: false, HasDefault: true}, {ID: 4, TypeID: 17, Null: false}, {ID: 5, TypeID: 17, Null: true}, {ID: 6, TypeID: 25, Null: true}, {ID: 7, TypeID: 20, Null: true}, {ID: 8, TypeID: 17, Null: true}, {ID: 9, TypeID: 20, Null: true}, {ID: 10, TypeID: 20, Null: true}, {ID: 11, TypeID: 1114, Null: true}], NextFamilyID: 3, Families: [{ID: 0, Columns: [1, 2, 3, 4, 6, 7]}, {ID: 1, Columns: [5]}, {ID: 2, Columns: [8, 9, 10, 11]}], PrimaryIndex: 1, NextIndexID: 5, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}, {ID: 2, Unique: false, KeyColumns: [{ID: 2, Dir: ASC}, {ID: 3, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 3, Unique: false, KeyColumns: [{ID: 6, Dir: ASC}, {ID: 7, Dir: ASC}], KeySuffixColumns: [1], StoreColumns: [2]}, {ID: 4, Unique: false, Partial: true, KeyColumns: [{ID: 8, Dir: ASC}, {ID: 2, Dir: ASC}, {ID: 3, Dir: ASC}], KeySuffixColumns: [1], StoreColumns: [11, 10, 9]}]}
```

It sure seems to take long breaks. Will assign to @cockroachdb/sql-experience to investigate further, as they seem to own the migration best I can tell.

I'm noticing is this recurring error message in tenant 11's logs:

> W210908 11:22:39.753402 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 â‹® [-] 124  TokenBucket error: â€¹tenant-usage-selectâ€º: descriptor not found

@RaduBerinde is that something to worry about or does the message's severity need to be reduced? Could this contribute to the slowdown?
 Comment 1: The message suggests that the tenant_usage table doesn't exist on the host, maybe something's not set up right in the test? In any case, we only recently (1-2 days ago) merged tenant-side throttling so there shouldn't have been any related slowdown in the test. 
 Comment 2: > maybe something's not set up right in the test?

The only nonstandard thing the tenant does is to upgrade the host cluster. Tenant 11 is created when the predecessor is running.

But I actually got something wrong in my initial analysis. I don't think tenant 11 matters here. The timeout is on the host cluster upgrade:

https://github.com/cockroachdb/cockroach/blob/642e44afbe6098f25022618be76c6f7b6b97df45/pkg/cmd/roachtest/tests/multitenant_upgrade.go#L321-L325

so I need to look at its logs.
 Comment 3: > 11:23:23 multitenant_upgrade.go:321: test status: finalizing the kv server

in the cockroach logs:

```
I210908 11:23:55.479745 12632 migration/migrationmanager/manager.go:115 â‹® [n1,intExec=â€¹set-versionâ€º,migration-mgr] 121  migrating cluster from 21.1-124 to 21.1-1160 (stepping through [21.1-1102 21.1-1104 21.1-1106 21.1-1108 21.1-1110 21.1-1112 21.1-1114 21.1-1116 21.1-1118 21.1-1120 21.1-1122 21.1-1124 21.1-1126 21.1-1128 21.1-1130 21.1-1132 21.1-1134 21.1-1136 21.1-1138 21.1-1140 21.1-1142 21.1-1144 21.1-1146 21.1-1148 21.1-1150 21.1-1152 21.1-1154 21.1-1156 21.1-1158 21.1-1160])
[...]
I210908 11:23:55.485586 12639 server/migration.go:149 â‹® [n1,bump-cluster-version] 125  active cluster version setting is now â€¹21.1-1101(fence) [...]
[...]
I210908 11:24:08.163833 17962 server/migration.go:149 â‹® [n1,bump-cluster-version] 495  active cluster version setting is now â€¹21.1-1160â€º (up from â€¹21.1-1159(fence)â€º)
```

the only slow movement there is from 1112 to 1113(fence), and it's not that slow - 9s. However, it seems to take 32s from the "test status" log line to first signs of the migration on the cluster. It's also relying on the auto upgrade loop here though, which has a sensible MaxBackoff of 30s, so this isn't surprising either:

https://github.com/cockroachdb/cockroach/blob/dbc8676679d21d3f61568d687e74b7c83329adb2/pkg/server/auto_upgrade.go#L32-L37

So I think we need to just bump how long the test is willing to wait.

I still think the tenant 11 logs merit a look from sql-experience as this migration does seem very slow in this case.

@RaduBerinde tenant 11 is upgraded before the host cluster, so the host cluster will jeremydixon@example.net and the tenant will run `master` and bump the cluster setting. (We haven't tagged master as 22.1 yet, we need a beta for that). So maybe there is some incompatibility?

 Comment 4: I looked at the migration, I couldn't reproduce it taking 5 minutes on master (even after changing the timeout back to 45 seconds)

The thing that stands out to me is that it looks like at time `I210908 11:18:30.169291` the descriptors get batch upgraded (which should end the migration).  And all the new leases are acquired by `11:18:30.207014` 
The remaining time (4ish minutes) only has the recurring error in the log
`W210908 11:23:19.753378 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 131  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:23:19.754046 110 2@util/log/event_log.go:32 ⋮ [-] 132 ={"Timestamp":1631100199754043860,"EventType":"runtime_stats","MemRSSBytes":165150720,"GoroutineCount":111,"MemStackSysBytes":1998848,"GoAllocBytes":33177376,"GoTotalBytes":65520648,"GoStatsStaleness":0.000782826,"HeapFragmentBytes":10625248,"HeapReservedBytes":10002432,"HeapReleasedBytes":11304960,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999947,"CPUUserPercent":0.7999958,"CPUSysPercent":0.19999894,"GCRunCount":15,"NetHostRecvBytes":166201,"NetHostSendBytes":99385}`

```
I210908 11:18:19.984511 309 migration/migrations/fix_descriptor_migration.go:108 ⋮ [sql,job=‹691501465340608513›,migration=21.1-1114] 61  upgrading descriptor with ids [1 3 4 6 7 8 11 12 13 14 15 19 20 21 23 24 25 26 27 28 31 32 33 34 35 36 37 39 40 50 51 52]
I210908 11:18:29.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 62 ={"Timestamp":1631099909740931541,"EventType":"runtime_stats","MemRSSBytes":145518592,"GoroutineCount":112,"MemStackSysBytes":2129920,"GoAllocBytes":28103064,"GoTotalBytes":52839432,"GoStatsStaleness":0.000837777,"HeapFragmentBytes":6762088,"HeapReservedBytes":6520832,"HeapReleasedBytes":23592960,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.00000005701674,"CPUUserPercent":0.000000033106495,"CPUSysPercent":0.000000007970082,"GCPausePercent":0.00000000012563811,"GCRunCount":9,"NetHostRecvBytes":242161,"NetHostSendBytes":149812}
W210908 11:18:29.752677 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 63  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:18:30.168223 309 sql/catalog/lease/lease.go:139 ⋮ [sql,job=‹691501465340608513›,migration=21.1-1114] 64  waiting for 1 leases to expire: desc=[{‹system› 1 1}]
I210908 11:18:30.169291 328 sql/catalog/lease/descriptor_state.go:139 ⋮ [-] 65  new lease: tabledesc.immutable: {ID: 6, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 5, Columns: [{ID: 1, TypeID: 25, Null: false}, {ID: 2, TypeID: 25, Null: false}, {ID: 3, TypeID: 1114, Null: false, HasDefault: true}, {ID: 4, TypeID: 25, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4]}], PrimaryIndex: 1, NextIndexID: 2, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4]}]}
I210908 11:18:30.176490 329 sql/catalog/lease/descriptor_state.go:139 ⋮ [-] 66  new lease: tabledesc.immutable: {ID: 15, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 10, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 25, Null: false}, {ID: 3, TypeID: 1114, Null: false, HasDefault: true}, {ID: 4, TypeID: 17, Null: false}, {ID: 5, TypeID: 17, Null: true}, {ID: 6, TypeID: 25, Null: true}, {ID: 7, TypeID: 20, Null: true}, {ID: 8, TypeID: 17, Null: true}, {ID: 9, TypeID: 20, Null: true}], NextFamilyID: 3, Families: [{ID: 0, Columns: [1, 2, 3, 4, 6, 7]}, {ID: 1, Columns: [5]}, {ID: 2, Columns: [8, 9]}], PrimaryIndex: 1, NextIndexID: 4, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7, 8, 9]}, {ID: 2, Unique: false, KeyColumns: [{ID: 2, Dir: ASC}, {ID: 3, Dir: ASC}], KeySuffixColumns: [1]}, {ID: 3, Unique: false, KeyColumns: [{ID: 6, Dir: ASC}, {ID: 7, Dir: ASC}], KeySuffixColumns: [1], StoreColumns: [2]}]}
I210908 11:18:30.182671 361 sql/catalog/lease/descriptor_state.go:139 ⋮ [-] 67  new lease: tabledesc.immutable: {ID: 20, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 10, Columns: [{ID: 1, TypeID: 20, Null: false}, {ID: 2, TypeID: 20, Null: false, HasDefault: true}, {ID: 3, TypeID: 25, Null: true}, {ID: 4, TypeID: 1016, Null: false}, {ID: 5, TypeID: 1114, Null: false, HasDefault: true}, {ID: 6, TypeID: 20, Null: false}, {ID: 7, TypeID: 20, Null: false}, {ID: 8, TypeID: 20, Null: false}, {ID: 9, TypeID: 17, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7, 8, 9]}], PrimaryIndex: 1, NextIndexID: 2, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}, {ID: 2, Dir: ASC}], StoreColumns: [3, 4, 5, 6, 7, 8, 9]}]}
I210908 11:18:30.190351 363 sql/catalog/lease/descriptor_state.go:139 ⋮ [-] 68  new lease: tabledesc.immutable: {ID: 35, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 6, Columns: [{ID: 1, TypeID: 20, Null: false, HasDefault: true}, {ID: 2, TypeID: 16, Null: false, HasDefault: true}, {ID: 3, TypeID: 25, Null: false}, {ID: 4, TypeID: 20, Null: true}, {ID: 5, TypeID: 1184, Null: false}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5]}], PrimaryIndex: 1, NextIndexID: 3, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5]}, {ID: 2, Unique: false, KeyColumns: [{ID: 2, Dir: ASC}, {ID: 1, Dir: ASC}], StoreColumns: [3]}]}
I210908 11:18:30.195894 366 sql/catalog/lease/descriptor_state.go:139 ⋮ [-] 69  new lease: tabledesc.immutable: {ID: 40, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 1, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 6, Columns: [{ID: 1, TypeID: 20, Null: false}, {ID: 2, TypeID: 20, Null: false}, {ID: 3, TypeID: 20, Null: false}, {ID: 4, TypeID: 20, Null: false}, {ID: 5, TypeID: 1184, Null: false}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5]}], PrimaryIndex: 1, NextIndexID: 2, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}, {ID: 2, Dir: ASC}, {ID: 3, Dir: ASC}, {ID: 4, Dir: ASC}], StoreColumns: [5]}]}
I210908 11:18:30.200513 337 sql/catalog/lease/descriptor_state.go:139 ⋮ [-] 70  new lease: dbdesc.immutable: {ID: 50, Version: 2, ModificationTime: "1631099910.082944016,1", State: PUBLIC}
I210908 11:18:30.207014 387 sql/catalog/lease/descriptor_state.go:139 ⋮ [-] 71  new lease: tabledesc.immutable: {ID: 52, Version: 2, ModificationTime: "1631099910.082944016,1", ParentID: 50, ParentSchemaID: 29, State: PUBLIC, NextColumnID: 3, Columns: [{ID: 1, TypeID: 20, Null: false}, {ID: 2, TypeID: 25, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2]}], PrimaryIndex: 1, NextIndexID: 2, Indexes: [{ID: 1, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2]}]}
I210908 11:18:39.741667 110 2@util/log/event_log.go:32 ⋮ [-] 72 ={"Timestamp":1631099919741662994,"EventType":"runtime_stats","MemRSSBytes":151089152,"GoroutineCount":110,"MemStackSysBytes":2064384,"GoAllocBytes":38650952,"GoTotalBytes":54395912,"GoStatsStaleness":0.000879517,"HeapFragmentBytes":3832760,"HeapReservedBytes":458752,"HeapReleasedBytes":22102016,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999269,"CPUUserPercent":1.7998685,"CPUSysPercent":0.2999781,"GCRunCount":9,"NetHostRecvBytes":156516,"NetHostSendBytes":81830}
W210908 11:18:39.755959 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 73  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:18:49.741845 110 2@util/log/event_log.go:32 ⋮ [-] 74 ={"Timestamp":1631099929741842360,"EventType":"runtime_stats","MemRSSBytes":153587712,"GoroutineCount":110,"MemStackSysBytes":2228224,"GoAllocBytes":26390152,"GoTotalBytes":57099272,"GoStatsStaleness":0.000754237,"HeapFragmentBytes":11063672,"HeapReservedBytes":8028160,"HeapReleasedBytes":19398656,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999819,"CPUUserPercent":1.0999801,"CPUSysPercent":0.39999276,"GCPausePercent":0.0013346858,"GCRunCount":10,"NetHostRecvBytes":274353,"NetHostSendBytes":47111}
W210908 11:18:49.752995 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 75  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:18:59.251322 48 sql/sqlliveness/slstorage/slstorage.go:376 ⋮ [-] 76  deleted 1 expired SQL liveness sessions
I210908 11:18:59.741937 110 2@util/log/event_log.go:32 ⋮ [-] 77 ={"Timestamp":1631099939741934089,"EventType":"runtime_stats","MemRSSBytes":153587712,"GoroutineCount":111,"MemStackSysBytes":2260992,"GoAllocBytes":29974624,"GoTotalBytes":57099272,"GoStatsStaleness":0.000782395,"HeapFragmentBytes":8232864,"HeapReservedBytes":7241728,"HeapReleasedBytes":19398656,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999909,"CPUUserPercent":0.89999187,"CPUSysPercent":0.1999982,"GCRunCount":10,"NetHostRecvBytes":80194,"NetHostSendBytes":44104}
W210908 11:18:59.753239 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 78  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:19:09.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 79 ={"Timestamp":1631099949742721725,"EventType":"runtime_stats","MemRSSBytes":153587712,"GoroutineCount":111,"MemStackSysBytes":2293760,"GoAllocBytes":33123888,"GoTotalBytes":57099272,"GoStatsStaleness":0.001064851,"HeapFragmentBytes":6287824,"HeapReservedBytes":6004736,"HeapReleasedBytes":19398656,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999213,"CPUUserPercent":0.79993707,"CPUSysPercent":0.19998427,"GCRunCount":10,"NetHostRecvBytes":267231,"NetHostSendBytes":40273}
W210908 11:19:09.753506 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 80  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:19:19.743064 110 2@util/log/event_log.go:32 ⋮ [-] 81 ={"Timestamp":1631099959743061683,"EventType":"runtime_stats","MemRSSBytes":153587712,"GoroutineCount":111,"MemStackSysBytes":2293760,"GoAllocBytes":36622160,"GoTotalBytes":57099272,"GoStatsStaleness":0.000820088,"HeapFragmentBytes":4772016,"HeapReservedBytes":4022272,"HeapReleasedBytes":19398656,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.0999966,"CPUUserPercent":0.69997615,"CPUSysPercent":0.3999864,"GCRunCount":10,"NetHostRecvBytes":167485,"NetHostSendBytes":94999}
W210908 11:19:19.752798 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 82  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:19:19.764333 504 jobs/registry.go:1150 ⋮ [sql] 83  AUTO CREATE STATS job ‹691501661402562561›: stepping through state running with error: <nil>
I210908 11:19:+1-205-728-7199 jobs/registry.go:1150 ⋮ [sql] 84  AUTO CREATE STATS job ‹691501661402562561›: stepping through state succeeded with error: <nil>
I210908 11:19:29.743144 110 2@util/log/event_log.go:32 ⋮ [-] 85 ={"Timestamp":1631099969743140082,"EventType":"runtime_stats","MemRSSBytes":155410432,"GoroutineCount":110,"MemStackSysBytes":1802240,"GoAllocBytes":22999408,"GoTotalBytes":58983432,"GoStatsStaleness":0.000734287,"HeapFragmentBytes":14814864,"HeapReservedBytes":9912320,"HeapReleasedBytes":17580032,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999921,"CPUUserPercent":1.5999874,"CPUSysPercent":0.29999763,"GCPausePercent":0.0018310455,"GCRunCount":11,"NetHostRecvBytes":106955,"NetHostSendBytes":60488}
W210908 11:19:29.753061 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 86  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:19:39.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 87 ={"Timestamp":1631099979743359355,"EventType":"runtime_stats","MemRSSBytes":155680768,"GoroutineCount":110,"MemStackSysBytes":1966080,"GoAllocBytes":26255144,"GoTotalBytes":58999816,"GoStatsStaleness":0.000726859,"HeapFragmentBytes":12067032,"HeapReservedBytes":9256960,"HeapReleasedBytes":17563648,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999781,"CPUUserPercent":0.7999824,"CPUSysPercent":0.1999956,"GCRunCount":11,"NetHostRecvBytes":77821,"NetHostSendBytes":41024}
W210908 11:19:39.753222 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 88  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:19:49.744076 110 2@util/log/event_log.go:32 ⋮ [-] 89 ={"Timestamp":1631099989744073224,"EventType":"runtime_stats","MemRSSBytes":155951104,"GoroutineCount":110,"MemStackSysBytes":1966080,"GoAllocBytes":29597648,"GoTotalBytes":58999816,"GoStatsStaleness":0.000730199,"HeapFragmentBytes":9412656,"HeapReservedBytes":8568832,"HeapReleasedBytes":17563648,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.099992864,"CPUUserPercent":0.8999358,"CPUSysPercent":0.19998574,"GCRunCount":11,"NetHostRecvBytes":272657,"NetHostSendBytes":43257}
W210908 11:19:49.753172 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 90  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:19:59.745636 110 2@util/log/event_log.go:32 ⋮ [-] 91 ={"Timestamp":1631099999745633205,"EventType":"runtime_stats","MemRSSBytes":155951104,"GoroutineCount":111,"MemStackSysBytes":1966080,"GoAllocBytes":33211512,"GoTotalBytes":58999816,"GoStatsStaleness":0.001041696,"HeapFragmentBytes":6863752,"HeapReservedBytes":7503872,"HeapReleasedBytes":17563648,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09998441,"CPUUserPercent":0.79987526,"CPUSysPercent":0.39993763,"GCRunCount":11,"NetHostRecvBytes":79591,"NetHostSendBytes":43251}
W210908 11:19:59.753026 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 92  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:20:09.746333 110 2@util/log/event_log.go:32 ⋮ [-] 93 ={"Timestamp":1631100009746330885,"EventType":"runtime_stats","MemRSSBytes":155951104,"GoroutineCount":111,"MemStackSysBytes":1966080,"GoAllocBytes":36528560,"GoTotalBytes":58999816,"GoStatsStaleness":0.000743596,"HeapFragmentBytes":4996688,"HeapReservedBytes":6053888,"HeapReleasedBytes":17563648,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999301,"CPUUserPercent":0.79994404,"CPUSysPercent":0.29997906,"GCRunCount":11,"NetHostRecvBytes":268324,"NetHostSendBytes":41316}
W210908 11:20:09.753138 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 94  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:20:19.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 95 ={"Timestamp":1631100019747245751,"EventType":"runtime_stats","MemRSSBytes":155951104,"GoroutineCount":111,"MemStackSysBytes":1998848,"GoAllocBytes":39694704,"GoTotalBytes":58999816,"GoStatsStaleness":0.000969086,"HeapFragmentBytes":3911312,"HeapReservedBytes":3940352,"HeapReleasedBytes":17563648,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999087,"CPUUserPercent":0.69993603,"CPUSysPercent":0.39996347,"GCRunCount":11,"NetHostRecvBytes":88266,"NetHostSendBytes":49015}
W210908 11:20:19.754542 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 96  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:20:29.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 97 ={"Timestamp":1631100029747892403,"EventType":"runtime_stats","MemRSSBytes":157401088,"GoroutineCount":110,"MemStackSysBytes":2097152,"GoAllocBytes":28947648,"GoTotalBytes":58999816,"GoStatsStaleness":0.001196554,"HeapFragmentBytes":13052736,"HeapReservedBytes":5447680,"HeapReleasedBytes":17563648,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999353,"CPUUserPercent":1.2999159,"CPUSysPercent":0.39997414,"GCPausePercent":0.0018272118,"GCRunCount":12,"NetHostRecvBytes":86779,"NetHostSendBytes":47015}
W210908 11:20:29.752409 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 98  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:20:39.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 99 ={"Timestamp":1631100039747523117,"EventType":"runtime_stats","MemRSSBytes":157896704,"GoroutineCount":110,"MemStackSysBytes":2293760,"GoAllocBytes":32322696,"GoTotalBytes":58999816,"GoStatsStaleness":0.000772671,"HeapFragmentBytes":10349432,"HeapReservedBytes":4579328,"HeapReleasedBytes":17563648,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.1000037,"CPUUserPercent":0.70002586,"CPUSysPercent":0.3000111,"GCRunCount":12,"NetHostRecvBytes":77529,"NetHostSendBytes":39700}
W210908 11:20:39.753134 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 100  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:20:49.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 101 ={"Timestamp":1631100049747699241,"EventType":"runtime_stats","MemRSSBytes":157896704,"GoroutineCount":110,"MemStackSysBytes":2260992,"GoAllocBytes":35680632,"GoTotalBytes":58999816,"GoStatsStaleness":0.000737496,"HeapFragmentBytes":7704200,"HeapReservedBytes":3899392,"HeapReleasedBytes":17563648,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999822,"CPUUserPercent":0.89998406,"CPUSysPercent":0.19999644,"GCRunCount":12,"NetHostRecvBytes":374593,"NetHostSendBytes":113147}
W210908 11:20:49.753899 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 102  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:20:59.748024 110 2@util/log/event_log.go:32 ⋮ [-] 103 ={"Timestamp":1631100059748020996,"EventType":"runtime_stats","MemRSSBytes":159076352,"GoroutineCount":111,"MemStackSysBytes":2293760,"GoAllocBytes":42627200,"GoTotalBytes":60736520,"GoStatsStaleness":0.000924011,"HeapFragmentBytes":4059008,"HeapReservedBytes":2301952,"HeapReleasedBytes":15826944,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999679,"CPUUserPercent":1.399955,"CPUSysPercent":0.29999036,"GCRunCount":12,"NetHostRecvBytes":125277,"NetHostSendBytes":73614}
W210908 11:20:59.752821 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 104  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:21:09.748339 110 2@util/log/event_log.go:32 ⋮ [-] 105 ={"Timestamp":1631100069748336470,"EventType":"runtime_stats","MemRSSBytes":161611776,"GoroutineCount":111,"MemStackSysBytes":1900544,"GoAllocBytes":25597960,"GoTotalBytes":62514184,"GoStatsStaleness":0.000973049,"HeapFragmentBytes":14821368,"HeapReservedBytes":10739712,"HeapReleasedBytes":14049280,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999685,"CPUUserPercent":1.0999653,"CPUSysPercent":0.49998426,"GCPausePercent":0.001301529,"GCRunCount":13,"NetHostRecvBytes":273436,"NetHostSendBytes":43519}
W210908 11:21:09.753073 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 106  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:21:19.748401 110 2@util/log/event_log.go:32 ⋮ [-] 107 ={"Timestamp":1631100079748397159,"EventType":"runtime_stats","MemRSSBytes":161611776,"GoroutineCount":111,"MemStackSysBytes":2031616,"GoAllocBytes":28753776,"GoTotalBytes":62604296,"GoStatsStaleness":0.001057417,"HeapFragmentBytes":12222608,"HeapReservedBytes":10141696,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.0999994,"CPUUserPercent":0.69999576,"CPUSysPercent":0.2999982,"GCRunCount":13,"NetHostRecvBytes":88693,"NetHostSendBytes":47111}
W210908 11:21:19.752657 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 108  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:21:29.748594 110 2@util/log/event_log.go:32 ⋮ [-] 109 ={"Timestamp":1631100089748591202,"EventType":"runtime_stats","MemRSSBytes":161611776,"GoroutineCount":110,"MemStackSysBytes":2031616,"GoAllocBytes":32771248,"GoTotalBytes":62604296,"GoStatsStaleness":0.000756409,"HeapFragmentBytes":8950608,"HeapReservedBytes":9396224,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999806,"CPUUserPercent":0.9999806,"CPUSysPercent":0.19999611,"GCRunCount":13,"NetHostRecvBytes":81302,"NetHostSendBytes":44557}
W210908 11:21:29.753130 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 110  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:21:39.749013 110 2@util/log/event_log.go:32 ⋮ [-] 111 ={"Timestamp":1631100099749009454,"EventType":"runtime_stats","MemRSSBytes":161611776,"GoroutineCount":110,"MemStackSysBytes":2031616,"GoAllocBytes":36055800,"GoTotalBytes":62604296,"GoStatsStaleness":0.000776627,"HeapFragmentBytes":6501640,"HeapReservedBytes":8560640,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.099995814,"CPUUserPercent":0.8999623,"CPUSysPercent":0.19999163,"GCRunCount":13,"NetHostRecvBytes":78441,"NetHostSendBytes":39868}
W210908 11:21:39.752807 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 112  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:21:49.749569 110 2@util/log/event_log.go:32 ⋮ [-] 113 ={"Timestamp":1631100109749566345,"EventType":"runtime_stats","MemRSSBytes":161611776,"GoroutineCount":110,"MemStackSysBytes":2031616,"GoAllocBytes":39326296,"GoTotalBytes":62604296,"GoStatsStaleness":0.000702881,"HeapFragmentBytes":4664744,"HeapReservedBytes":7127040,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999442,"CPUUserPercent":0.89994985,"CPUSysPercent":0.19998886,"GCRunCount":13,"NetHostRecvBytes":272432,"NetHostSendBytes":45294}
W210908 11:21:49.753363 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 114  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:21:59.750332 110 2@util/log/event_log.go:32 ⋮ [-] 115 ={"Timestamp":1631100119750329943,"EventType":"runtime_stats","MemRSSBytes":161873920,"GoroutineCount":111,"MemStackSysBytes":2031616,"GoAllocBytes":42918480,"GoTotalBytes":62604296,"GoStatsStaleness":0.000730253,"HeapFragmentBytes":3898800,"HeapReservedBytes":4300800,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.099992365,"CPUUserPercent":0.89993125,"CPUSysPercent":0.19998473,"GCRunCount":13,"NetHostRecvBytes":81322,"NetHostSendBytes":42694}
W210908 11:21:59.754463 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 116  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:22:09.750905 110 2@util/log/event_log.go:32 ⋮ [-] 117 ={"Timestamp":1631100129750902586,"EventType":"runtime_stats","MemRSSBytes":162193408,"GoroutineCount":111,"MemStackSysBytes":1998848,"GoAllocBytes":29700472,"GoTotalBytes":62604296,"GoStatsStaleness":0.000776577,"HeapFragmentBytes":14855816,"HeapReservedBytes":6594560,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999428,"CPUUserPercent":1.2999256,"CPUSysPercent":0.29998282,"GCPausePercent":0.0018744528,"GCRunCount":14,"NetHostRecvBytes":268360,"NetHostSendBytes":41805}
W210908 11:22:09.753170 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 118  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:22:19.751053 110 2@util/log/event_log.go:32 ⋮ [-] 119 ={"Timestamp":1631100139751050356,"EventType":"runtime_stats","MemRSSBytes":162193408,"GoroutineCount":111,"MemStackSysBytes":2195456,"GoAllocBytes":33058312,"GoTotalBytes":62604296,"GoStatsStaleness":0.000827926,"HeapFragmentBytes":12095992,"HeapReservedBytes":5799936,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999852,"CPUUserPercent":0.6999896,"CPUSysPercent":0.3999941,"GCRunCount":14,"NetHostRecvBytes":92036,"NetHostSendBytes":50908}
W210908 11:22:19.752991 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 120  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:22:29.752010 110 2@util/log/event_log.go:32 ⋮ [-] 121 ={"Timestamp":1631100149752007239,"EventType":"runtime_stats","MemRSSBytes":162193408,"GoroutineCount":110,"MemStackSysBytes":2260992,"GoAllocBytes":37317544,"GoTotalBytes":62604296,"GoStatsStaleness":0.000896951,"HeapFragmentBytes":8664152,"HeapReservedBytes":4907008,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999044,"CPUUserPercent":0.899914,"CPUSysPercent":0.29997134,"GCRunCount":14,"NetHostRecvBytes":83900,"NetHostSendBytes":45758}
W210908 11:22:29.753576 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 122  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:22:39.752175 110 2@util/log/event_log.go:32 ⋮ [-] 123 ={"Timestamp":1631100159752171598,"EventType":"runtime_stats","MemRSSBytes":162193408,"GoroutineCount":110,"MemStackSysBytes":2293760,"GoAllocBytes":40616656,"GoTotalBytes":62604296,"GoStatsStaleness":0.000734363,"HeapFragmentBytes":6274352,"HeapReservedBytes":3964928,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.099998355,"CPUUserPercent":0.89998525,"CPUSysPercent":0.29999506,"GCRunCount":14,"NetHostRecvBytes":78782,"NetHostSendBytes":41644}
W210908 11:22:39.753402 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 124  TokenBucket error: ‹tenant-usage-select›: descriptor not found
W210908 11:22:49.752668 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 125  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:22:49.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 126 ={"Timestamp":1631100169753347244,"EventType":"runtime_stats","MemRSSBytes":162414592,"GoroutineCount":110,"MemStackSysBytes":2260992,"GoAllocBytes":43872840,"GoTotalBytes":62866440,"GoStatsStaleness":0.000928772,"HeapFragmentBytes":4500920,"HeapReservedBytes":2514944,"HeapReleasedBytes":13959168,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.099988244,"CPUUserPercent":0.79990596,"CPUSysPercent":0.19997649,"GCRunCount":14,"NetHostRecvBytes":275770,"NetHostSendBytes":44233}
W210908 11:22:59.753053 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 127  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:22:59.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 128 ={"Timestamp":1631100179753697140,"EventType":"runtime_stats","MemRSSBytes":164880384,"GoroutineCount":111,"MemStackSysBytes":1900544,"GoAllocBytes":26429928,"GoTotalBytes":65299464,"GoStatsStaleness":0.000801152,"HeapFragmentBytes":16061976,"HeapReservedBytes":11190272,"HeapReleasedBytes":11526144,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999649,"CPUUserPercent":1.2999544,"CPUSysPercent":0.2999895,"GCPausePercent":0.0011267904,"GCRunCount":15,"NetHostRecvBytes":80937,"NetHostSendBytes":43351}
W210908 11:23:09.753255 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 129  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:23:09.+1-205-728-7199@util/log/event_log.go:32 ⋮ [-] 130 ={"Timestamp":1631100189753991306,"EventType":"runtime_stats","MemRSSBytes":165150720,"GoroutineCount":111,"MemStackSysBytes":1998848,"GoAllocBytes":29880112,"GoTotalBytes":65520648,"GoStatsStaleness":0.000819486,"HeapFragmentBytes":13242576,"HeapReservedBytes":10682368,"HeapReleasedBytes":11304960,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999706,"CPUUserPercent":0.99997056,"CPUSysPercent":0.19999412,"GCRunCount":15,"NetHostRecvBytes":294712,"NetHostSendBytes":56551}
W210908 11:23:19.753378 140 ccl/multitenantccl/tenantcostclient/tenant_side.go:143 ⋮ [-] 131  TokenBucket error: ‹tenant-usage-select›: descriptor not found
I210908 11:23:19.754046 110 2@util/log/event_log.go:32 ⋮ [-] 132 ={"Timestamp":1631100199754043860,"EventType":"runtime_stats","MemRSSBytes":165150720,"GoroutineCount":111,"MemStackSysBytes":1998848,"GoAllocBytes":33177376,"GoTotalBytes":65520648,"GoStatsStaleness":0.000782826,"HeapFragmentBytes":10625248,"HeapReservedBytes":10002432,"HeapReleasedBytes":11304960,"CGoAllocBytes":1072096,"CGoTotalBytes":2306048,"CGoCallRate":0.09999947,"CPUUserPercent":0.7999958,"CPUSysPercent":0.19999894,"GCRunCount":15,"NetHostRecvBytes":166201,"NetHostSendBytes":99385}
I210908 11:23:20.756226 309 jobs/registry.go:1150 ⋮ [sql] 133  MIGRATION job ‹691501465340608513›: stepping through state succeeded with error: <nil>
I210908 11:23:21.482004 214 migration/migrationmanager/manager.go:135 ⋮ [sql,client=‹116.69.203.115:44454›,hostnossl,user=root,migration-mgr] 134  stepping through 21.1-1116
```
 Comment 5: roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/viewLog.html?buildId=3555378&tab=buildLog) with [artifacts](https://teamcity.cockroachdb.com/viewLog.html?buildId=3555378&tab=artifacts#/multitenant-upgrade) on master @ [7e4ba61845bb47cc2d7146d1bbd70fd53eff5457](https://github.com/cockroachdb/cockroach/commits/7e4ba61845bb47cc2d7146d1bbd70fd53eff5457):


```
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2042
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2021
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1942
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:119
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:114
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:294
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) cluster.RunL
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
```
<details><summary>Reproduce</summary>
<p>


See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

</p>
</details>
<details><summary>Same failure on other branches</summary>
<p>

- #71269 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot branch-release-21.2]
</p>
</details>
/cc @cockroachdb/kv-triage
<sub>

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)
</sub>

 Comment 6: roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/viewLog.html?buildId=3560297&tab=buildLog) with [artifacts](https://teamcity.cockroachdb.com/viewLog.html?buildId=3560297&tab=artifacts#/multitenant-upgrade) on master @ [fb67c7f24a05673bf0bb06f72a4c697679d1fff4](https://github.com/cockroachdb/cockroach/commits/fb67c7f24a05673bf0bb06f72a4c697679d1fff4):


```
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2042
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2021
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1942
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:119
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:114
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:294
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) cluster.RunL
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
```
<details><summary>Reproduce</summary>
<p>


See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

</p>
</details>
<details><summary>Same failure on other branches</summary>
<p>

- #71269 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot branch-release-21.2]
</p>
</details>
/cc @cockroachdb/kv-triage
<sub>

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)
</sub>

 Comment 7: roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/viewLog.html?buildId=3562413&tab=buildLog) with [artifacts](https://teamcity.cockroachdb.com/viewLog.html?buildId=3562413&tab=artifacts#/multitenant-upgrade) on master @ [afb9f8dd5c4b66c39a15dc16e1b9ca07c09816bd](https://github.com/cockroachdb/cockroach/commits/afb9f8dd5c4b66c39a15dc16e1b9ca07c09816bd):


```
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2042
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2021
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1942
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:119
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:114
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:294
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) cluster.RunL
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
```
<details><summary>Reproduce</summary>
<p>


See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

</p>
</details>
<details><summary>Same failure on other branches</summary>
<p>

- #71269 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot branch-release-21.2]
</p>
</details>
/cc @cockroachdb/kv-triage
<sub>

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)
</sub>

 Comment 8: roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/viewLog.html?buildId=3570227&tab=buildLog) with [artifacts](https://teamcity.cockroachdb.com/viewLog.html?buildId=3570227&tab=artifacts#/multitenant-upgrade) on master @ [de5c5d344e6c27b29d0ac9e5503a762aad67fb52](https://github.com/cockroachdb/cockroach/commits/de5c5d344e6c27b29d0ac9e5503a762aad67fb52):


```
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2042
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2021
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1942
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:119
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:114
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:294
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) cluster.RunL
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
```
<details><summary>Reproduce</summary>
<p>


See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

</p>
</details>
<details><summary>Same failure on other branches</summary>
<p>

- #71269 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot branch-release-21.2]
</p>
</details>
/cc @cockroachdb/kv-triage
<sub>

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)
</sub>

 Comment 9: I guess the recent flakiness. I'm phobic that we haven't been tracking skipped and flaky tests with nearly the zeal of earlier releases. I clicked through the email and commented out of panic given the whole upgrade debacle yesterday. Sorry for the noise given your active investigation. What is interesting about "first bad build" is that it has PRs which rely on the borked migration we fixed yesterday with https://github.com/cockroachdb/cockroach/pull/71492. Perhaps this test was a clue we had a problem.

For what it's worth, that issue was egregious and would have popped out immediately if we actually ever looked at a cluster which we upgraded. I miss the days of `cyan`. 
 Comment 10: I looked into this more just now and I don't blame the `pkill` any more. The test reliably fails when it's trying to restart tenant 11 at the new version. I've inserted an artificial `time.Sleep(15*time.Second)` before the `start` operation and am still seeing the issue. I've also run `watch -n 1 "ps auxwww | grep mt-11"` on the side and the tenant process shuts down pretty much when it says it does (i.e. at the beginning of the 15s window). And yet, the same error occurs (the temp directory exists and looks like a pebble dir, just as it should). At the same time, when I try to reproduce the failing start invocation by running the test with `--debug` and then invoking

```
/home/tobias/go/src/github.com/cockroachdb/cockroach/artifacts/5cd42f90c7/roachprod run local:2 -- ./cockroach mt start-sql --insecure --tenant-id=11 --http-addr 116.69.203.115:8011 --kv-addrs 116.69.203.115:26257 --sql-addr 116.69.203.115:20011 --log-dir=logs/mt-11
```

manually, the tenant server appears to get past that point.
 Comment 11: roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/viewLog.html?buildId=3593048&tab=buildLog) with [artifacts](https://teamcity.cockroachdb.com/viewLog.html?buildId=3593048&tab=artifacts#/multitenant-upgrade) on master @ [b3af96b0686773c78325d2b8b0623a8fcd3e9bf2](https://github.com/cockroachdb/cockroach/commits/b3af96b0686773c78325d2b8b0623a8fcd3e9bf2):


```
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2042
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2021
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1942
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:119
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:114
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:294
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) cluster.RunL
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
```
<details><summary>Reproduce</summary>
<p>


See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

</p>
</details>
<details><summary>Same failure on other branches</summary>
<p>

- #71269 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot branch-release-21.2]
</p>
</details>
/cc @cockroachdb/kv-triage
<sub>

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)
</sub>

 Comment 12: roachtest.multitenant-upgrade [failed](https://teamcity.cockroachdb.com/viewLog.html?buildId=3595417&tab=buildLog) with [artifacts](https://teamcity.cockroachdb.com/viewLog.html?buildId=3595417&tab=artifacts#/multitenant-upgrade) on master @ [5d972a683c531326bc0af403c6d3373d3f4b2267](https://github.com/cockroachdb/cockroach/commits/5d972a683c531326bc0af403c6d3373d3f4b2267):


```
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2042
		  | main.(*clusterImpl).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2021
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1942
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | main.(*clusterImpl).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1944
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:84
		  | runtime.gopanic
		  | 	/usr/local/go/src/runtime/panic.go:965
		  | main.(*testImpl).fatalfInner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:284
		  | main.(*testImpl).Fatal
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:119
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:114
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:294
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:39
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (4) cluster.RunL
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString
```
<details><summary>Reproduce</summary>
<p>


See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

</p>
</details>
<details><summary>Same failure on other branches</summary>
<p>

- #71269 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot branch-release-21.2]
</p>
</details>
/cc @cockroachdb/kv-triage
<sub>

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*multitenant-upgrade.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)
</sub>

{'source': 'GithubIssues', 'repo_name_with_owner': 'cockroachdb/cockroach', 'issue_id': 991029482, 'repo_stars': 25990.0}