<s> [INST] <<SYS>>
These are Failed Build Logs. You have to find the Build Failure Causing Errors. Ignore the errors irrelevant to the build failure <</SYS>>

2024-02-02 22:09:23 - INFO-root::util|431:: #9 0.364 # get https://proxy.golang.org/github.com/go-errors/errors/@v/v1.4.2.mod[0m
2024-02-02 22:09:23 - INFO-root::util|431:: #9 0.366 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.19.2.mod[0m
2024-02-02 22:09:24 - INFO-root::util|431:: #9 1.440 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v1.1.1.mod[0m
2024-02-02 22:09:24 - INFO-root::util|431:: #9 1.448 # get https://proxy.golang.org/github.com/go-errors/errors/@v/v1.4.2.mod: 200 OK (1.084s)[0m
2024-02-02 22:09:24 - INFO-root::util|431:: #9 1.459 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.19.2.mod: 200 OK (1.093s)[0m
2024-02-02 22:09:24 - INFO-root::util|431:: #9 1.665 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.9.1.mod[0m
2024-02-02 22:09:24 - INFO-root::util|431:: #9 1.677 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v1.1.1.mod: 200 OK (0.237s)[0m
2024-02-02 22:09:24 - INFO-root::util|431:: #9 1.900 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.9.1.mod: 200 OK (0.235s)[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 2.374 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.8.1.mod[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 2.421 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20200804184101-5ec99f83aff1.mod[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 2.430 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v1.0.0.mod[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 2.582 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.18.0.mod[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 2.603 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.8.1.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 2.650 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20200804184101-5ec99f83aff1.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 2.658 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v1.0.0.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 2.811 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.18.0.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:25 - INFO-root::util|431:: #9 3.047 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.17.0.mod[0m
2024-02-02 22:09:26 - INFO-root::util|431:: #9 3.261 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20191011141410-1b5146add898.mod[0m
2024-02-02 22:09:26 - INFO-root::util|431:: #9 3.266 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20191204190536-9bdfabe68543.mod[0m
2024-02-02 22:09:26 - INFO-root::util|431:: #9 3.275 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.17.0.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:26 - INFO-root::util|431:: #9 3.490 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20191011141410-1b5146add898.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:26 - INFO-root::util|431:: #9 3.495 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20191204190536-9bdfabe68543.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:26 - INFO-root::util|431:: #9 3.503 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20190717185122-a985d3407aa7.mod[0m
2024-02-02 22:09:26 - INFO-root::util|431:: #9 3.733 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20190717185122-a985d3407aa7.mod: 200 OK (0.230s)[0m
2024-02-02 22:09:26 - INFO-root::util|431:: #9 4.046 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.8.0.mod[0m
2024-02-02 22:09:27 - INFO-root::util|431:: #9 4.275 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.8.0.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:28 - INFO-root::util|431:: #9 5.571 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20190513163551-3ee3066db522.mod[0m
2024-02-02 22:09:28 - INFO-root::util|431:: #9 5.805 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20190513163551-3ee3066db522.mod: 200 OK (0.235s)[0m
2024-02-02 22:09:29 - INFO-root::util|431:: #9 6.955 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.8.1-0.20171018195549-f15c970de5b7.mod[0m
2024-02-02 22:09:29 - INFO-root::util|431:: #9 6.980 # get https://proxy.golang.org/github.com/cockroachdb/errors/@v/v1.2.4.mod[0m
2024-02-02 22:09:29 - INFO-root::util|431:: #9 7.028 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20190410155217-1f06c39b4373.mod[0m
2024-02-02 22:09:29 - INFO-root::util|431:: #9 7.171 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v0.0.0-20161216184304-ed905158d874.mod[0m
2024-02-02 22:09:30 - INFO-root::util|431:: #9 7.184 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.8.1-0.20171018195549-f15c970de5b7.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:30 - INFO-root::util|431:: #9 7.209 # get https://proxy.golang.org/github.com/cockroachdb/errors/@v/v1.2.4.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:30 - INFO-root::util|431:: #9 7.257 # get https://proxy.golang.org/golang.org/x/xerrors/@v/v0.0.0-20190410155217-1f06c39b4373.mod: 200 OK (0.229s)[0m
2024-02-02 22:09:30 - INFO-root::util|431:: #9 7.401 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v0.0.0-20161216184304-ed905158d874.mod: 200 OK (0.230s)[0m
2024-02-02 22:09:38 - INFO-root::util|431:: #9 15.77 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.9.1.info[0m
2024-02-02 22:09:38 - INFO-root::util|431:: #9 16.00 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.9.1.info: 200 OK (0.229s)[0m
2024-02-02 22:09:55 - INFO-root::util|431:: #9 32.48 # get https://proxy.golang.org/github.com/go-errors/errors/@v/v1.4.2.info[0m
2024-02-02 22:09:55 - INFO-root::util|431:: #9 32.71 # get https://proxy.golang.org/github.com/go-errors/errors/@v/v1.4.2.info: 200 OK (0.232s)[0m
2024-02-02 22:09:56 - INFO-root::util|431:: #9 33.88 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.19.2.info[0m
2024-02-02 22:09:56 - INFO-root::util|431:: #9 34.11 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.19.2.info: 200 OK (0.233s)[0m
2024-02-02 22:10:04 - INFO-root::util|431:: #9 41.56 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v1.1.1.info[0m
2024-02-02 22:10:04 - INFO-root::util|431:: #9 41.80 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v1.1.1.info: 200 OK (0.232s)[0m
2024-02-02 22:10:34 - INFO-root::util|431:: #9 72.13 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.9.1.zip[0m
2024-02-02 22:10:35 - INFO-root::util|431:: #9 73.01 # get https://proxy.golang.org/github.com/pkg/errors/@v/v0.9.1.zip: 200 OK (0.876s)[0m
2024-02-02 22:10:35 - INFO-root::util|431:: #9 73.01 # get https://proxy.golang.org/github.com/go-errors/errors/@v/v1.4.2.zip[0m
2024-02-02 22:10:35 - INFO-root::util|431:: #9 73.12 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.19.2.zip[0m
2024-02-02 22:10:36 - INFO-root::util|431:: #9 73.49 # get https://proxy.golang.org/github.com/go-errors/errors/@v/v1.4.2.zip: 200 OK (0.484s)[0m
2024-02-02 22:10:36 - INFO-root::util|431:: #9 73.56 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v1.1.1.zip[0m
2024-02-02 22:10:36 - INFO-root::util|431:: #9 73.56 # get https://proxy.golang.org/github.com/go-openapi/errors/@v/v0.19.2.zip: 200 OK (0.444s)[0m
2024-02-02 22:10:36 - INFO-root::util|431:: #9 74.00 # get https://proxy.golang.org/github.com/hashicorp/go-multierror/@v/v1.1.1.zip: 200 OK (0.442s)[0m
2024-02-02 22:18:51 - [31mERROR-root::util|431:: Error response from daemon: No such container: service-test-dsPAxKO8[0m
2024-02-02 22:20:48 - INFO-root::util|431:: === RUN   TestNodeListVisitor_Visit/Nil_data_returns_error[0m
2024-02-02 22:20:48 - INFO-root::util|431::     --- PASS: TestNodeListVisitor_Visit/Nil_data_returns_error (0.00s)[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === RUN   TestAWSBackupClient/WaitForRestoreJobs/job_failed[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === PAUSE TestAWSBackupClient/WaitForRestoreJobs/job_failed[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === CONT  TestAWSBackupClient/WaitForRestoreJobs/job_failed[0m
2024-02-02 22:20:51 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:643","message":"[requestId=req-id] restore job j1 FAILED. Additional message: some additional msg","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:643","message":"[requestId=req-id] restore job j2 FAILED. Additional message: some additional msg","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:643","message":"[requestId=req-id] restore job j3 FAILED. Additional message: some additional msg","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j1. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j2. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j2. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j2. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j3. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j3. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:631","message":"[requestId=req-id] failed to describe restore job j3. Error: unexpected error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431::         --- PASS: TestAWSBackupClient/WaitForRestoreJobs/job_failed (0.00s)[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === RUN   TestDescribeRestoreJobs/Failed_Descriptions[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === PAUSE TestDescribeRestoreJobs/Failed_Descriptions[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === CONT  TestDescribeRestoreJobs/Failed_Descriptions[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:744","message":"[requestId=req-id] failed to describe restore job j1. Error: description error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:744","message":"[requestId=req-id] failed to describe restore job j2. Error: description error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"aws/aws_backup.go:744","message":"[requestId=req-id] failed to describe restore job j3. Error: description error","requestId":"req-id","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:20:51 - INFO-root::util|431::     --- PASS: TestDescribeRestoreJobs/Failed_Descriptions (0.00s)[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === RUN   TestLoadBalancerClient/DescribeV2LoadBalancers/error[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === PAUSE TestLoadBalancerClient/DescribeV2LoadBalancers/error[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === CONT  TestLoadBalancerClient/DescribeV2LoadBalancers/error[0m
2024-02-02 22:20:51 - INFO-root::util|431::         --- PASS: TestLoadBalancerClient/DescribeV2LoadBalancers/error (0.00s)[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === RUN   TestGetDefaultKey/ListAliases_errored[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === PAUSE TestGetDefaultKey/ListAliases_errored[0m
2024-02-02 22:20:51 - INFO-root::util|431:: === CONT  TestGetDefaultKey/ListAliases_errored[0m
2024-02-02 22:20:51 - INFO-root::util|431::     --- PASS: TestGetDefaultKey/ListAliases_errored (0.00s)[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestPrivateCloudBackupClient/GetOrCreateBackupVault_should_return_error_for_private_cloud[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestPrivateCloudBackupClient/GetRecoveryPointsFromBackupJobs_should_return_error_if_drs_backup_job_fails[0m
2024-02-02 22:21:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"backup/backup_private.go:194","message":"[requestId=baba3060-da60-4735-aab5-52849bdceff3] backup job drsBackupJobCrn FAILED","requestId":"baba3060-da60-4735-aab5-52849bdceff3","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431::     --- PASS: TestPrivateCloudBackupClient/GetOrCreateBackupVault_should_return_error_for_private_cloud (0.00s)[0m
2024-02-02 22:21:21 - INFO-root::util|431::     --- PASS: TestPrivateCloudBackupClient/GetRecoveryPointsFromBackupJobs_should_return_error_if_drs_backup_job_fails (0.00s)[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflowPrivate/Test_Workflow/PVC_Workspace_(in_installed:finished_state)_should_recover_if_there_was_error_in_scaling_down_[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflowPrivate/Test_Workflow/PVC_Workspace_(in_installed:finished_state)_should_recover_if_there_was_error_in_starting_storage_backup_jobs[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"level":"info","ts":1706912450.4098923,"caller":"workflow_utils/utils.go:77","msg":"Worflow was canceled, details 'unexpected error'","requestId":"08424268-9ce5-4de7-a6ea-015f64edce69","actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","resourceId":"crn:cdp:ml:us-west-1:1111111:workspace_backup:881688a9-c320-4ab0-bae3-b3a630e5c345"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflowPrivate/Test_Activities/Test_CreateAggregatedPVCPodActivity/CreateAggregatedPVCPodActivity_should_not_throw_error_if_the_pod_already_exist_[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflowPrivate/Test_Activities/Test_DeleteAggregatedPVCPodActivity/DeleteAggregatedPVCPodActivity_should_not_throw_error_if_the_pod_was_already_deleted_somehow_[0m
2024-02-02 22:21:21 - INFO-root::util|431::         --- PASS: TestWorkspaceBackupWorkflowPrivate/Test_Workflow/PVC_Workspace_(in_installed:finished_state)_should_recover_if_there_was_error_in_scaling_down_ (0.01s)[0m
2024-02-02 22:21:21 - INFO-root::util|431::         --- PASS: TestWorkspaceBackupWorkflowPrivate/Test_Workflow/PVC_Workspace_(in_installed:finished_state)_should_recover_if_there_was_error_in_starting_storage_backup_jobs (0.01s)[0m
2024-02-02 22:21:21 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflowPrivate/Test_Activities/Test_CreateAggregatedPVCPodActivity/CreateAggregatedPVCPodActivity_should_not_throw_error_if_the_pod_already_exist_ (0.00s)[0m
2024-02-02 22:21:21 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflowPrivate/Test_Activities/Test_DeleteAggregatedPVCPodActivity/DeleteAggregatedPVCPodActivity_should_not_throw_error_if_the_pod_was_already_deleted_somehow_ (0.00s)[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Workflow/error_in_scaling_down[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Workflow/error_during_starting_storage_backup_jobs[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"level":"info","ts":1706912450.4575994,"caller":"workflow_utils/utils.go:77","msg":"Worflow was canceled, details 'unexpected error'","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","resourceId":"crn:cdp:ml:us-west-1:1111111:workspace_backup:727e3773-adb8-4f04-9728-69628c627688"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_AddBackupEntryActivity/helm_values_get_failure[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_AddBackupEntryActivity/creating_DB_entry_failure[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_ScaleUpReplicasActivity/workspace_not_in_suspend:finished_status/error_in_StartDeploymentsAsync_and_StartStatefulsets[0m
2024-02-02 22:21:21 - INFO-root::util|431:: 2024/02/02 22:20:50 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"backup/backup_workflow.go:1873","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] unexpected error
unexpected error
","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: 2024/02/02 22:20:50 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"RESTART_AFTER_BACKUP_FAILED","OldStatus":"INSTALLATION_FINISHED","WorkspaceCrn":"crn:cdp:ml:us-west-1:1111111:workspace:f12b3a5e-6d1d-4bb2-aca7-3350b9da434d"}'","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"backup/backup_workflow.go:1132","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] post-processing failed. Error: unexpected error
unexpected error
","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_ScaleUpReplicasActivity/workspace_not_in_suspend:finished_status/error_in_WaitforDeployment[0m
2024-02-02 22:21:21 - INFO-root::util|431:: 2024/02/02 22:20:50 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"backup/backup_workflow.go:1888","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] error waiting for deployments to come up: unexpected error","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: 2024/02/02 22:20:50 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"RESTART_AFTER_BACKUP_FAILED","OldStatus":"INSTALLATION_FINISHED","WorkspaceCrn":"crn:cdp:ml:us-west-1:1111111:workspace:f12b3a5e-6d1d-4bb2-aca7-3350b9da434d"}'","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"backup/backup_workflow.go:1132","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] post-processing failed. Error: unexpected error","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: 2024/02/02 22:20:50 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: 2024/02/02 22:20:50 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_RetrieveStorageIDsActivity/error_getting_block_volumes[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_StartStorageBackupJobsActivity/error_starting_block_volume_jobs[0m
2024-02-02 22:21:21 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_StartStorageBackupJobsActivity/error_starting_file_system_job[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_DetermineResourcesToScaleDownActivity/error_getting_statefulsets[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_DetermineResourcesToScaleDownActivity/error_getting_deployments[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_DetermineResourcesToScaleDownActivity/error_getting_cron_deployment[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"warning","line":"backup/backup_workflow.go:1591","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] error in fetching deployment cron: unexpected error","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestWorkspaceBackupWorkflow/Test_Activities/Test_ScaleDownReplicasActivity/error_shutting_down_deployments_and_statefulsets[0m
2024-02-02 22:21:22 - INFO-root::util|431:: 2024/02/02 22:21:20 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: 2024/02/02 22:21:20 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] {"id":"f4101092-670a-4b02-981c-2dc65c325308","timestamp":1706912480512,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:1111111:workspace:f12b3a5e-6d1d-4bb2-aca7-3350b9da434d","oldStatus":5,"newStatus":41,"failureReason":"unexpected error\nunexpected error\n","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","liftieId":"liftie-xx","userTags":"{}","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"SHUTDOWN_FOR_BACKUP_FAILED","OldStatus":"INSTALLATION_FINISHED","WorkspaceCrn":"crn:cdp:ml:us-west-1:1111111:workspace:f12b3a5e-6d1d-4bb2-aca7-3350b9da434d"}'","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: 2024/02/02 22:21:20 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: 2024/02/02 22:21:20 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-xx  Continuing... Get "/liftie/api/v1/cluster/liftie-xx": unsupported protocol scheme ""","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::         --- PASS: TestWorkspaceBackupWorkflow/Test_Workflow/error_in_scaling_down (0.01s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::         --- PASS: TestWorkspaceBackupWorkflow/Test_Workflow/error_during_starting_storage_backup_jobs (0.01s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_AddBackupEntryActivity/helm_values_get_failure (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_AddBackupEntryActivity/creating_DB_entry_failure (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::                 --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_ScaleUpReplicasActivity/workspace_not_in_suspend:finished_status/error_in_StartDeploymentsAsync_and_StartStatefulsets (0.01s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::                 --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_ScaleUpReplicasActivity/workspace_not_in_suspend:finished_status/error_in_WaitforDeployment (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_RetrieveStorageIDsActivity/error_getting_block_volumes (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_StartStorageBackupJobsActivity/error_starting_block_volume_jobs (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_StartStorageBackupJobsActivity/error_starting_file_system_job (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_DetermineResourcesToScaleDownActivity/error_getting_statefulsets (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_DetermineResourcesToScaleDownActivity/error_getting_deployments (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_DetermineResourcesToScaleDownActivity/error_getting_cron_deployment (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestWorkspaceBackupWorkflow/Test_Activities/Test_ScaleDownReplicasActivity/error_shutting_down_deployments_and_statefulsets (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestBackupWorkspaceCancellationWorkflow/Test_Activties/Test_CancelStorageBackupJobsActivity/error_in_cancelling_backup_jobs[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"warning","line":"backup/cancellation_workflows.go:293","message":"[requestId=918ba582-5772-44d4-a735-f04ae0ddc2a1] attempted to cancel backup jobs '[j1 j2]'. Encountered error: unexpected error ","requestId":"918ba582-5772-44d4-a735-f04ae0ddc2a1","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"warning","line":"backup/cancellation_workflows.go:293","message":"[requestId=918ba582-5772-44d4-a735-f04ae0ddc2a1] attempted to cancel backup jobs '[j1]'. Encountered error: <nil> ","requestId":"918ba582-5772-44d4-a735-f04ae0ddc2a1","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestBackupWorkspaceCancellationWorkflow/Test_Activties/Test_CancelStorageBackupJobsActivity/error_in_cancelling_backup_jobs (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestDeleteBackupWorkflow/Full_Workflow_Tests/all_activties_in_workflow_passes_without_error[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=df329716-2bf6-40a4-b4aa-ba5be9ff7b94] {"id":"c97c5b77-7000-47f1-b88a-4128d8ae9914","timestamp":1706912480544,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"backup-crn","oldStatus":4,"newStatus":6,"failureReason":"unexpected error","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94"}}}","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: {"level":"error","ts":1706912480.5454628,"caller":"backup/backup_workflow.go:2184","msg":"error in cleaning up backup snapshot resources. Error: unexpected error","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94","actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","resourceId":"backup-crn","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup.(*DeleteBackupWorkflow).RunDeleteBackupWorkflow
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow.go:2184
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup_test.TestDeleteBackupWorkflow.func1.2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow_test.go:1220
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestDeleteBackupWorkflow/Full_Workflow_Tests/ClearBackupVault_activity_failed[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=df329716-2bf6-40a4-b4aa-ba5be9ff7b94] {"id":"31ef0328-907f-4b64-9eba-ee7832b6c608","timestamp":1706912480551,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"backup-crn","oldStatus":4,"newStatus":6,"failureReason":"unexpected error","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94"}}}","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: {"level":"error","ts":1706912480.5520027,"caller":"backup/backup_workflow.go:2189","msg":"error in clearing backup vault. Error: unexpected error","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94","actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","resourceId":"backup-crn","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup.(*DeleteBackupWorkflow).RunDeleteBackupWorkflow
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow.go:2189
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup_test.TestDeleteBackupWorkflow.func1.3
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow_test.go:1247
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestDeleteBackupWorkflow/Full_Workflow_Tests/UpdateBackupFieldsInDatabase_activity_failed[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: {"level":"error","ts":1706912480.558084,"caller":"backup/backup_workflow.go:2209","msg":"error in updating/soft-deleting backup db entry. Error: unexpected error","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94","actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","resourceId":"backup-crn","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup.(*DeleteBackupWorkflow).RunDeleteBackupWorkflow
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow.go:2209
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup_test.TestDeleteBackupWorkflow.func1.4
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow_test.go:1274
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=df329716-2bf6-40a4-b4aa-ba5be9ff7b94] {"id":"4ecdfbad-4319-4fd3-9ddd-e619c0ccaff4","timestamp":1706912480566,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"backup-crn","oldStatus":4,"newStatus":6,"failureReason":"unexpected error","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94"}}}","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94","timestamp":"2024-02-02T22:21:20Z"}[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: {"level":"error","ts":1706912480.5677588,"caller":"backup/backup_workflow.go:2189","msg":"error in clearing backup vault. Error: unexpected error","requestId":"df329716-2bf6-40a4-b4aa-ba5be9ff7b94","actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","resourceId":"backup-crn","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup.(*DeleteBackupWorkflow).RunDeleteBackupWorkflow
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow.go:2189
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup_test.TestDeleteBackupWorkflow.func1.5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow_test.go:1303
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestDeleteBackupWorkflow/Test_Activities/Test_CleanBackupResources/error_in_checking_for_backup_vault[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: 2024-02-02T22:21:20.580Z	ERROR	backup/backup_workflow.go:2236	error in checking for backup vault. Error: random error in checking for backup vault	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "req-id", "actorCrn": "actor-crn", "resourceId": "backup-crn"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestDeleteBackupWorkflow/Test_Activities/Test_CleanBackupResources/error_in_resource_clean_up[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: 2024-02-02T22:21:20.582Z	ERROR	backup/backup_workflow.go:2242	error in deleting backup resources. Error: random error in removing resources	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "req-id", "actorCrn": "actor-crn", "resourceId": "backup-crn"}[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: 2024-02-02T22:21:20.584Z	ERROR	backup/backup_workflow.go:2264	error in checking for backup vault. Error: random error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "req-id", "actorCrn": "actor-crn", "resourceId": "backup-crn"}[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: 2024-02-02T22:21:20.586Z	ERROR	backup/backup_workflow.go:2270	error in checking if backup vault is empty. Error: random error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "req-id", "actorCrn": "actor-crn", "resourceId": "backup-crn"}[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: 2024-02-02T22:21:20.588Z	ERROR	backup/backup_workflow.go:2275	error deleting backup vault. Error: random error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "req-id", "actorCrn": "actor-crn", "resourceId": "backup-crn"}[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: 2024-02-02T22:21:20.589Z	ERROR	backup/backup_workflow.go:2295	error updating backup status in DB. Error: random error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "req-id", "actorCrn": "actor-crn", "resourceId": "backup-crn"}[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: 2024-02-02T22:21:20.591Z	ERROR	backup/backup_workflow.go:2300	error marking the backup as deleted. Error: random error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "req-id", "actorCrn": "actor-crn", "resourceId": "backup-crn"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::         --- PASS: TestDeleteBackupWorkflow/Full_Workflow_Tests/all_activties_in_workflow_passes_without_error (0.01s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::         --- PASS: TestDeleteBackupWorkflow/Full_Workflow_Tests/ClearBackupVault_activity_failed (0.01s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::         --- PASS: TestDeleteBackupWorkflow/Full_Workflow_Tests/UpdateBackupFieldsInDatabase_activity_failed (0.01s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestDeleteBackupWorkflow/Test_Activities/Test_CleanBackupResources/error_in_checking_for_backup_vault (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::             --- PASS: TestDeleteBackupWorkflow/Test_Activities/Test_CleanBackupResources/error_in_resource_clean_up (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestModelRegistryBackupWorkflow/Test_Workflow/error_in_scaling_down[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestModelRegistryBackupWorkflow/Test_Workflow/error_during_starting_storage_backup_jobs[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"level":"error","ts":1706912480.6311553,"caller":"backup/backup_workflow.go:433","msg":"Failure Reason: WorkflowCancellationRequestedError","requestId":"8f990b23-2be5-4b01-bf1c-b81c5f8dc397","actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","resourceId":"crn:cdp:ml:us-west-1:1111111:model_registry_backup:59a98d52-7dc1-49f3-8b1d-d6a9db3f254d","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup.(*ModelRegistryBackupWorkflow).RunWorkflow
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow.go:433
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup_test.TestModelRegistryBackupWorkflow.func1.5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/backup/backup_workflow_test.go:1776
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"level":"info","ts":1706912480.6330342,"caller":"workflow_utils/utils.go:77","msg":"Worflow was canceled, details 'unexpected error'","requestId":"8f990b23-2be5-4b01-bf1c-b81c5f8dc397","actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","resourceId":"crn:cdp:ml:us-west-1:1111111:model_registry_backup:59a98d52-7dc1-49f3-8b1d-d6a9db3f254d"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::         --- PASS: TestModelRegistryBackupWorkflow/Test_Workflow/error_in_scaling_down (0.01s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::         --- PASS: TestModelRegistryBackupWorkflow/Test_Workflow/error_during_starting_storage_backup_jobs (0.01s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetDefaultCMLChartVersion/DB_errors_out[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === PAUSE TestGetDefaultCMLChartVersion/DB_errors_out[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === CONT  TestGetDefaultCMLChartVersion/DB_errors_out[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=046fc8e0-82f5-4950-b568-58a7662b004b] could not obtain default workload version from configuration variables in DB","requestId":"046fc8e0-82f5-4950-b568-58a7662b004b","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetDefaultCMLChartVersion/DB_errors_out (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetInstallableCMLVersions/DB_query_errors_out[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=df6c7bba-287b-4f74-a959-9def7ebdc56f] could not obtain default workload version from configuration variables in DB","requestId":"df6c7bba-287b-4f74-a959-9def7ebdc56f","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=df6c7bba-287b-4f74-a959-9def7ebdc56f] could not obtain default workload version from configuration variables in DB","requestId":"df6c7bba-287b-4f74-a959-9def7ebdc56f","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetInstallableCMLVersions/DB_query_errors_out (0.00s)[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetBlockVolumesUsedByWorkspaceAWS/Error_in_getting_namespace_with_AWS[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:121","message":"[requestId=test-uuid] Failed to get namespace: failed to get namespace","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetBlockVolumesUsedByWorkspaceAWS/Error_in_getting_PVCs_with_AWS[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:128","message":"[requestId=test-uuid] Failed to get PVCs: failed to get PVCs","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetBlockVolumesUsedByWorkspaceAWS/Error_in_getting_EBS_volumes_from_AWS[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:190","message":"[requestId=test-uuid] Failed to get EBS volumes: failed to get EBS volumes from AWS","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetBlockVolumesUsedByWorkspaceAWS/Error_in_getting_namespace_with_AWS (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetBlockVolumesUsedByWorkspaceAWS/Error_in_getting_PVCs_with_AWS (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetBlockVolumesUsedByWorkspaceAWS/Error_in_getting_EBS_volumes_from_AWS (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetBlockVolumesUsedByWorkspaceAzure/Error_in_getting_cloud_credentials_for_Azure[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:463","message":"[requestId=test-uuid] Failed to get cloud credentials: failed to get cloud credentials","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetBlockVolumesUsedByWorkspaceAzure/Error_in_getting_environment_details_for_Azure[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:487","message":"[requestId=test-uuid] Failed to fetch environment details: failed to fetch environment details","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetBlockVolumesUsedByWorkspaceAzure/Error_in_getting_complimentary_resource_group_for_Azure[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:496","message":"[requestId=test-uuid] Failed to get complimentary resource group for AKS with liftie ID test-liftie-id and region test-region","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestGetBlockVolumesUsedByWorkspaceAzure/Error_in_getting_disks_for_Azure[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:502","message":"[requestId=test-uuid] Failed to get disks for AKS cluster: failed to get disks for AKS cluster","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetBlockVolumesUsedByWorkspaceAzure/Error_in_getting_cloud_credentials_for_Azure (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetBlockVolumesUsedByWorkspaceAzure/Error_in_getting_environment_details_for_Azure (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetBlockVolumesUsedByWorkspaceAzure/Error_in_getting_complimentary_resource_group_for_Azure (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestGetBlockVolumesUsedByWorkspaceAzure/Error_in_getting_disks_for_Azure (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestTagBlockVolumesUsedByWorkspacesAWS/Error_in_tagging_volumes_with_AWS[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"warning","line":"cloudapicontroller/cloud_api_controller.go:220","message":"[requestId=test-uuid] WARNING: Failed to tag volume 'vol-12345678' for PVC 'failed to tag volumes'. Error: 123","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"warning","line":"cloudapicontroller/cloud_api_controller.go:220","message":"[requestId=test-uuid] WARNING: Failed to tag volume 'vol-23456789' for PVC 'failed to tag volumes'. Error: 123","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestTagBlockVolumesUsedByWorkspacesAWS/Error_in_tagging_volumes_with_AWS (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestTagBlockVolumesUsedByWorkspacesAzure/Error_in_getting_cloud_credentials_for_Azure[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:522","message":"[requestId=test-uuid] Failed to get cloud credentials: failed to get cloud credentials","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestTagBlockVolumesUsedByWorkspacesAzure/Error_when_tagging_Azure_disk[0m
2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"warning","line":"cloudapicontroller/cloud_api_controller.go:542","message":"[requestId=test-uuid] WARNING: Failed to tag volume 'test-volume-id' for workspace ID 123. Error: tagging error","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestTagBlockVolumesUsedByWorkspacesAzure/Error_in_getting_cloud_credentials_for_Azure (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestTagBlockVolumesUsedByWorkspacesAzure/Error_when_tagging_Azure_disk (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestTagNFSUsedByWorkspaceAWS/Error_in_tagging_EFS_with_AWS[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:245","message":"[requestId=test-uuid] Failed to tag EFS: failed to tag EFS","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestTagNFSUsedByWorkspaceAWS/Error_in_tagging_EFS_with_AWS (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestStartNFSRestoreJobAWS/Error_in_starting_EFS_restore_job_with_AWS[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:275","message":"[requestId=test-uuid] Failed to start EFS restore job: failed to start EFS restore job","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestStartNFSRestoreJobAWS/Error_in_starting_EFS_restore_job_with_AWS (0.00s)[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestStartBlockVolumesRestoreJobAWS/Error_in_starting_EBS_restore_jobs_with_AWS[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:309","message":"[requestId=test-uuid] Failed to start EBS restore jobs: failed to start EBS restore jobs","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431::     --- PASS: TestStartBlockVolumesRestoreJobAWS/Error_in_starting_EBS_restore_jobs_with_AWS (0.00s)[0m
2024-02-02 22:21:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:358","message":"[requestId=test-uuid] Restore job job-2 has status FAILED. ","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:22 - INFO-root::util|431:: === RUN   TestAreRestoreJobsCompletedAWS/Error_in_describing_restore_jobs_with_AWS[0m
2024-02-02 22:21:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:338","message":"[requestId=test-uuid] Failed to describe restore jobs: failed to describe restore jobs","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestAreRestoreJobsCompletedAWS/Error_in_describing_restore_jobs_with_AWS (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestListRestoreVolumeInfoAWS/Failed_mapping_due_to_missing_PVC[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:411","message":"[requestId=test-uuid] Not able to fetch restore job descriptions for all PVCs.","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestListRestoreVolumeInfoAWS/Error_in_describing_restore_jobs_with_AWS[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:384","message":"[requestId=test-uuid] Failed to describe restore jobs: failed to describe restore jobs","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestListRestoreVolumeInfoAWS/Failed_mapping_due_to_missing_PVC (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestListRestoreVolumeInfoAWS/Error_in_describing_restore_jobs_with_AWS (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestStartBlockVolumesRestoreJobAzure/Error_getting_Liftie_ID[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestStartBlockVolumesRestoreJobAzure/Error_getting_backup_catalog[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:587","message":"[requestId=test-uuid] Error while getting backup catalog for CRN 'test-backup-crn': Backup catalog not found","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestStartBlockVolumesRestoreJobAzure/Error_getting_environment_details[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:601","message":"[requestId=test-uuid] Error while getting environment details for environment name 'test-environment': Error fetching environment details","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestStartBlockVolumesRestoreJobAzure/Error_getting_Liftie_ID (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestStartBlockVolumesRestoreJobAzure/Error_getting_backup_catalog (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestStartBlockVolumesRestoreJobAzure/Error_getting_environment_details (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestAreRestoreJobsCompletedAzure/At_least_one_restore_job_failed[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cloudapicontroller/cloud_api_controller.go:682","message":"[requestId=test-uuid] Disk 'disk-2' is in a failed state","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestAreRestoreJobsCompletedAzure/Error_describing_Azure_disks[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"Failed to describe Azure disks","level":"error","line":"cloudapicontroller/cloud_api_controller.go:669","message":"[requestId=test-uuid] Failed to describe Azure disks","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestAreRestoreJobsCompletedAzure/At_least_one_restore_job_failed (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestAreRestoreJobsCompletedAzure/Error_describing_Azure_disks (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestListRestoreVolumeInfoAzure/Azure_disk_description_error[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"Failed to describe Azure disks","level":"error","line":"cloudapicontroller/cloud_api_controller.go:721","message":"[requestId=test-uuid] Failed to describe Azure disks","requestId":"test-uuid","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestListRestoreVolumeInfoAzure/Azure_disk_description_error (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestAuditEventLogs/Verify_cluster_timeout_writes_failed_events_to_context_events[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/cluster.go:1135","message":"[requestId=test1234] Failure event ProvisionerEvent Mock Name: Cluster liftieID-1, resource ProvisionerEvent Mock ResourceType has status ProvisionerEvent Mock Status for reason ProvisionerEvent Mock StatusReason. request-id: test1234
","requestId":"test1234","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestAuditEventLogs/Verify_cluster_timeout_writes_failed_events_to_context_events (0.26s)[0m
2024-02-02 22:21:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/cluster.go:1738","message":"[requestId=test1234] liftie pre-flight validation response: message: '[code: 1102, msg: error validating cluster creation request] info: could not parse CIDR test: invalid CIDR address: test","requestId":"test1234","timestamp":"2024-02-02T22:20:50Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestCreateClusterValidation/test_create_cluster_validator_returns_400_status_code_and_'_in_error_response[0m
2024-02-02 22:21:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/cluster.go:1738","message":"[requestId=test1234] liftie pre-flight validation response: message: '[code: 1102, msg: error validating cluster creation request] info: when specifying the subnet(s) to use, a virtual network must also be supplied via the environment'\n","requestId":"test1234","timestamp":"2024-02-02T22:20:51Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestCreateClusterValidation/test_create_cluster_validator_returns_400_status_code_and_'_in_error_response (0.18s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"info","line":"cluster/cluster.go:2857","message":"[requestId=5bafe43f-cc5d-48d5-9871-441df429bf7c] Current cluster status:  UPGRADE_FAILED","requestId":"5bafe43f-cc5d-48d5-9871-441df429bf7c","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"info","line":"cluster/cluster.go:2857","message":"[requestId=5bafe43f-cc5d-48d5-9871-441df429bf7c] Current cluster status:  UPGRADE_FAILED","requestId":"5bafe43f-cc5d-48d5-9871-441df429bf7c","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"info","line":"cluster/cluster.go:2857","message":"[requestId=5bafe43f-cc5d-48d5-9871-441df429bf7c] Current cluster status:  UPGRADE_FAILED","requestId":"5bafe43f-cc5d-48d5-9871-441df429bf7c","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"info","line":"cluster/cluster.go:2857","message":"[requestId=5bafe43f-cc5d-48d5-9871-441df429bf7c] Current cluster status:  UPGRADE_FAILED","requestId":"5bafe43f-cc5d-48d5-9871-441df429bf7c","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"info","line":"cluster/cluster.go:2857","message":"[requestId=5bafe43f-cc5d-48d5-9871-441df429bf7c] Current cluster status:  UPGRADE_FAILED","requestId":"5bafe43f-cc5d-48d5-9871-441df429bf7c","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestUpgradeKubernetesIterative_MocksRetries/1_error_and_5_upgrades_return_success[0m
2024-02-02 22:21:23 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"warning","line":"cluster/cluster.go:2889","message":"[requestId=e4271fe6-6b37-4bdb-b7e1-fbb6ebc3236f] Failed to upgrade Kubernetes:  mock error . Will retry again in 10 seconds...","requestId":"e4271fe6-6b37-4bdb-b7e1-fbb6ebc3236f","timestamp":"2024-02-02T22:20:52Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestUpgradeKubernetesIterative_MocksRetries/6_continuous_upgrades_request_return_error[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestUpgradeKubernetesIterative_MocksRetries/1_error_and_5_upgrades_return_success (10.01s)[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestUpgradeKubernetesIterative_MocksRetries/6_continuous_upgrades_request_return_error (0.01s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/cluster.go:3025","message":"[requestId=test1234] install csi-driver-nfs response: 500 : message : release not found","requestId":"test1234","timestamp":"2024-02-02T22:21:02Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/cluster.go:3124","message":"[requestId=test1234] update aws-efs-csi-driver response: 500 : message : release not found","requestId":"test1234","timestamp":"2024-02-02T22:21:02Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/cluster.go:3139","message":"[requestId=test1234] Failed to install CSI drivers. error : install aws-efs-csi-driver response: 500 : message : release not found","requestId":"test1234","timestamp":"2024-02-02T22:21:02Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/cluster.go:1053","message":"[requestId=returns-500] status code: 500; response body: ","requestId":"returns-500","timestamp":"2024-02-02T22:21:02Z"}[0m
2024-02-02 22:21:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/cluster.go:1040","message":"[requestId=returns-error] error in receiving response from liftie: Get "/liftie/api/v1/cluster?tenant=testAccountID&env=fakeEnvironmentCrn&workloads=ml&page=1&status=RUNNING": this should be a timeout","requestId":"returns-error","timestamp":"2024-02-02T22:21:02Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/crp_compute.go:429","message":"[requestId=3f4d93d0-80fd-401d-b229-e344a34e5e16] Get Cluster ID Response: "resp_parse_failed"","requestId":"3f4d93d0-80fd-401d-b229-e344a34e5e16","timestamp":"2024-02-02T22:21:02Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestUpdateResourcePools/test_the_EnableUmbraIntegration_enabled,_UseDefaultResourcePool_enabled,_GetResourcePool_error_out[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestUpdateResourcePools/test_the_EnableUmbraIntegration_enabled,_UseDefaultResourcePool_enabled,_GetResourcePool_not_error_out[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestUpdateResourcePools/test_the_EnableUmbraIntegration_enabled,_UseDefaultResourcePool_enabled,_GetResourcePool_error_out (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestUpdateResourcePools/test_the_EnableUmbraIntegration_enabled,_UseDefaultResourcePool_enabled,_GetResourcePool_not_error_out (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestDspBillOfMaterials_GetBillOfMaterials/Test_timeout_error[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestDspBillOfMaterials_GetBillOfMaterials/Test_error_when_unmarshalling_response[0m
2024-02-02 22:21:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/dsp_bom.go:106","message":"[requestId=unit-test-Test error when unmarshalling response] Dsp-bom service response 'This should fail'","requestId":"unit-test-Test error when unmarshalling response","timestamp":"2024-02-02T22:21:02Z"}[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestDspBillOfMaterials_GetBillOfMaterials/Test_timeout_error (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestDspBillOfMaterials_GetBillOfMaterials/Test_error_when_unmarshalling_response (0.00s)[0m
2024-02-02 22:21:23 - INFO-root::util|431:: === RUN   TestDspBillOfMaterials_GetBillOfMaterialsForK8sVersion/Test_timeout_error[0m
2024-02-02 22:21:23 - INFO-root::util|431::     --- PASS: TestDspBillOfMaterials_GetBillOfMaterialsForK8sVersion/Test_timeout_error (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestCrudImplInstall_InstallWorkspace/Test_install_workflow_not_started_when_Cb.GetEnvironment_errors[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-account:environment:test-env","level":"error","line":"core/core_install.go:56","message":"[requestId=5e23f8c6-410b-45cb-879b-236d56289698] Failed to get environment info. Reason: GetEnvironment error","requestId":"5e23f8c6-410b-45cb-879b-236d56289698","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-account:environment:test-env","level":"warning","line":"core/core.go:526","message":"[requestId=0abec07d-fcd1-4a31-afa1-57bc5f0bbcff] The preflight check for Cloudbreak failed:  ","requestId":"0abec07d-fcd1-4a31-afa1-57bc5f0bbcff","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-account:environment:test-env","level":"error","line":"core/core_install.go:76","message":"[requestId=c9cccec6-9e2f-4128-8a7f-e0af30153243] No support for cloud provider 'GCP'","requestId":"c9cccec6-9e2f-4128-8a7f-e0af30153243","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestCrudImplInstall_InstallWorkspace/Test_install_workflow_not_started_when_PEM.GetDomainName_errors[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-account:environment:test-env","level":"error","line":"core/core_install.go:536","message":"[requestId=c7341087-2214-4538-97f8-f6424825f5a1] GetDomainName error","requestId":"c7341087-2214-4538-97f8-f6424825f5a1","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-account:environment:test-env","level":"error","line":"core/core_install.go:123","message":"[requestId=c7341087-2214-4538-97f8-f6424825f5a1] failed to generate domains for workspaceName 'cml-mlx-liftie-1sgxwwnn', environmentName 'eng-ml-dev-env-aws', cmlExternalUniqueId 'ml-22d9d5bc-4f6': GetDomainName error","requestId":"c7341087-2214-4538-97f8-f6424825f5a1","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestCrudImplInstall_InstallWorkspace/Test_install_workflow_not_started_when_Cb.GetEnvironment_errors (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestCrudImplInstall_InstallWorkspace/Test_install_workflow_not_started_when_PEM.GetDomainName_errors (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestCrudImplInstall_startInstallWorkflow/Workflow_start_failure_is_reported[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestCrudImplInstall_startInstallWorkflow/Usage_reporting_error_causes_failure[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestCrudImplInstall_startInstallWorkflow/Db.AddEvent_error_causes_failure[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-account:environment:test-env","level":"error","line":"core/core_install.go:231","message":"[requestId=b5b0bdad-a114-4671-b51e-9596e9ae9655] Failed to AddEvent for workspaceCrn '', operation 'installation', operationStatus 'started': Db.AddEvent error","requestId":"b5b0bdad-a114-4671-b51e-9596e9ae9655","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestCrudImplInstall_startInstallWorkflow/Workflow_start_failure_is_reported (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestCrudImplInstall_startInstallWorkflow/Usage_reporting_error_causes_failure (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestCrudImplInstall_startInstallWorkflow/Db.AddEvent_error_causes_failure (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestCrudImpl_RunPreInstallValidations/error_handler_should_be_called_when_validations_throws_error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"error","line":"core/core.go:4528","message":"[requestId=a11e5f8b-983a-42ec-a6f9-d72b6ff375f4] Failed to validate pre-install checks for workspace: validations failed
","requestId":"a11e5f8b-983a-42ec-a6f9-d72b6ff375f4","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestCrudImpl_RunPreInstallValidations/error_handler_should_be_called_when_validations_fail[0m
2024-02-02 22:22:14 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"info","line":"core/core.go:4628","message":"[requestId=0cfaf718-ee8b-4de4-ad76-3a5cc28c43a6] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 0, Skipped: 0, Total: 0","requestId":"0cfaf718-ee8b-4de4-ad76-3a5cc28c43a6","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"error","line":"core/core.go:4528","message":"[requestId=0cfaf718-ee8b-4de4-ad76-3a5cc28c43a6] Failed to validate pre-install checks for workspace: Pre-install validations failed due to failures: '{"failedValidations":[]}'.
","requestId":"0cfaf718-ee8b-4de4-ad76-3a5cc28c43a6","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestCrudImpl_RunPreInstallValidations/error_handler_should_be_called_when_validations_throws_error (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestCrudImpl_RunPreInstallValidations/error_handler_should_be_called_when_validations_fail (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: unexpected error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: 2024-02-02T22:21:07.541Z	ERROR	core/create_workspace_workflow.go:134	Failed to TrackWorkspaceStatusChange 'provision:failed' due to error 'unexpected error'	{"requestId": "request-id", "actorCrn": "crn:altus:iam:us-west-1:9d74eee4-7ccf9edbb73d:user:a325601f-d52a-4cd49fbesdas", "resourceId": "ut-1"}[0m
2024-02-02 22:22:14 - INFO-root::util|431:: Failed to create Computex cluster: unexpected error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: Failed to create Computex cluster: unexpected error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: 2024-02-02T22:21:07.552Z	ERROR	core/create_workspace_workflow.go:322	Failed to TrackWorkspaceStatusChange 'installation:finished' due to error 'unexpected error'	{"requestId": "request-id", "actorCrn": "crn:altus:iam:us-west-1:9d74eee4-7ccf9edbb73d:user:a325601f-d52a-4cd49fbesdas", "resourceId": "ut-1"}[0m
2024-02-02 22:22:14 - INFO-root::util|431:: Failed to track the workspace status changes: unexpected error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: Failed to install workspace: unexpected error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_AreAllMountTargetsReadyActivity/returns_error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_AreAllMountTargetsReadyActivity/existingNfsServer_with_AWS_causes_error[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_AreAllMountTargetsReadyActivity/returns_error (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_AreAllMountTargetsReadyActivity/existingNfsServer_with_AWS_causes_error (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckAllContainersAreReadyActivity/getKubeConfig_throws_error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckAllContainersAreReadyActivity/GetAdminKubeConfig_throws_error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckAllContainersAreReadyActivity/List_pod_throws_error[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckAllContainersAreReadyActivity/getKubeConfig_throws_error (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckAllContainersAreReadyActivity/GetAdminKubeConfig_throws_error (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckAllContainersAreReadyActivity/List_pod_throws_error (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/GetAdminKubeConfig_throws_error[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_unable_to_GetReleaseStatus_errors[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus_is_nil[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info_is_nil[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info.Status_==_release.StatusFailed[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info.Status_==_release.StatusDeployed_and_GetReleaseStatus.Info.Description_!=_Install_complete[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/GetAdminKubeConfig_throws_error (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_unable_to_GetReleaseStatus_errors (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus_is_nil (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info_is_nil (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info.Status_==_release.StatusFailed (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info.Status_==_release.StatusDeployed_and_GetReleaseStatus.Info.Description_!=_Install_complete (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CreateAccessPointActivity/returns_error_when_access_point_creation_request_fails[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"level":"error","ts":1706912467.6080227,"caller":"core/install_activities.go:212","msg":"Failed to create access point for workspaceCrn 'workspaceCrn' due to error 'access point creation failed'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).CreateAccessPointActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:212
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_CreateAccessPointActivity.func3
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:824
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CreateAccessPointActivity/returns_error_when_access_point_creation_request_fails (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CreateStorageMountTargetActivity/returns_error_after_creating_mount_targets_fail[0m
2024-02-02 22:22:14 - INFO-root::util|431:: {"level":"error","ts":1706912467.6124985,"caller":"core/install_activities.go:199","msg":"Failed to create mount target for workspaceCrn 'workspace-crn', fsId 'ml-1234', storageType 'EFS': failed","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).CreateStorageMountTargetActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:199
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_CreateStorageMountTargetActivity.func6
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:969
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CreateStorageMountTargetActivity/returns_error_when_existingNfsServer_is_provided_for_AWS[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_CreateStorageMountTargetActivity/returns_error_when_unable_to_determine_mount_target_status[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CreateStorageMountTargetActivity/returns_error_after_creating_mount_targets_fail (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CreateStorageMountTargetActivity/returns_error_when_existingNfsServer_is_provided_for_AWS (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_CreateStorageMountTargetActivity/returns_error_when_unable_to_determine_mount_target_status (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_DeleteHelmReleaseActivity/returns_error_when_GetAdminKubeConfig_errors[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_DeleteHelmReleaseActivity/returns_error_when_helm.DeleteUsingSDK_fails[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_DeleteHelmReleaseActivity/returns_error_when_GetAdminKubeConfig_errors (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_DeleteHelmReleaseActivity/returns_error_when_helm.DeleteUsingSDK_fails (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_DoesHelmReleaseExistActivity/returns_error_when_GetAdminKubeConfig_errors[0m
2024-02-02 22:22:14 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_DoesHelmReleaseExistActivity/returns_false_when_GetReleaseStatus_throws_error[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_DoesHelmReleaseExistActivity/returns_error_when_GetAdminKubeConfig_errors (0.00s)[0m
2024-02-02 22:22:14 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_DoesHelmReleaseExistActivity/returns_false_when_GetReleaseStatus_throws_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_InstallCMLWorkspaceActivity/returns_error_when_helm_install_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:364","message":"[requestId=request-id] Failed helm install: InstallAsync error
","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:365","message":"[requestId=request-id] helm stdout: 
","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:366","message":"[requestId=request-id] helm stderr: 
","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912467.8953226,"caller":"core/install_activities.go:753","msg":"Installation failed: 'InstallAsync error'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).InstallCMLWorkspaceActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:753
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_InstallCMLWorkspaceActivity.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:1667
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_InstallCMLWorkspaceActivity/returns_error_when_GetCDSWChartPath_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912467.898032,"caller":"core/install_activities.go:753","msg":"Installation failed: 'GetCDSWChartPath error'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).InstallCMLWorkspaceActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:753
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_InstallCMLWorkspaceActivity.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:1667
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_InstallCMLWorkspaceActivity/returns_error_when_GetAdminKubeConfig_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_InstallCMLWorkspaceActivity/returns_error_when_helm_install_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_InstallCMLWorkspaceActivity/returns_error_when_GetCDSWChartPath_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_InstallCMLWorkspaceActivity/returns_error_when_GetAdminKubeConfig_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_InstallCSIDriversActivity/returns_error_when_ClusterProvisioner.InstallCSIDrivers_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:tenant1234:user:user1234","level":"info","line":"core/core_install.go:479","message":"[requestId=request-id] Failed to install CSI drivers. error : InstallCSIDrivers error","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912467.9056072,"caller":"core/install_activities.go:115","msg":"InstallCSIDriver failed for workspaceCrn 'workspace-123': InstallCSIDrivers error","requestId":"request-id","actorCrn":"crn:altus:iam:us-west-1:tenant1234:user:user1234","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).InstallCSIDriversActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:115
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_InstallCSIDriversActivity.func3
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:1750
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_InstallCSIDriversActivity/returns_error_when_ClusterProvisioner.InstallCSIDrivers_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_IsAccessPointReadyActivity/returns_error[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_IsAccessPointReadyActivity/returns_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_IsFileSystemReadyActivity/returns_error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_IsFileSystemReadyActivity/returns_error_when_existingNfsServer_is_provided_for_AWS[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_IsFileSystemReadyActivity/returns_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_IsFileSystemReadyActivity/returns_error_when_existingNfsServer_is_provided_for_AWS (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_PatchDnsInPublicCloudActivity/returns_error_when_GetAdminKubeConfig_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_PatchDnsInPublicCloudActivity/returns_error_when_GetCloudSession_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_PatchDnsInPublicCloudActivity/returns_error_when_RegisterDnsV2_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"info","line":"core/core_install.go:388","message":"[requestId=request-id] Failed to register dns for : RegisterDnsV2 error
","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_PatchDnsInPublicCloudActivity/returns_error_when_CreateFreeIpaDnsEntry_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:393","message":"[requestId=request-id] failed to create DNS entry in FreeIpa: CreateFreeIpaDnsEntry error","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_PatchDnsInPublicCloudActivity/returns_error_when_GetAdminKubeConfig_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_PatchDnsInPublicCloudActivity/returns_error_when_GetCloudSession_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_PatchDnsInPublicCloudActivity/returns_error_when_RegisterDnsV2_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_PatchDnsInPublicCloudActivity/returns_error_when_CreateFreeIpaDnsEntry_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_PostInstallActivity/returns_error_when_GetAdminKubeConfig_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_PostInstallActivity/returns_error_when_PostInstallV2_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"info","ts":1706912467.924149,"caller":"core/install_activities.go:597","msg":"Failed to complete post install operations: PostInstallV2 error
","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id"}[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_PostInstallActivity/returns_error_when_GetAdminKubeConfig_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_PostInstallActivity/returns_error_when_PostInstallV2_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_LaunchPreInstallValidationsActivity/returns_error_when_pre_install_validations_gets_launch_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_LaunchPreInstallValidationsActivity/returns_error_when_pre_install_validations_gets_launch_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912467.955609,"caller":"core/install_activities.go:312","msg":"PreInstallCheckValidator has elapsed maximum wait time of '15ms'. Activity will now stop heartbeating","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).ValidatePreInstallChecksActivity.func1
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:312"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_ValidatePreInstallChecksActivity/returns_error_when_validations_fail[0m
2024-02-02 22:22:15 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"info","line":"core/core.go:4628","message":"[requestId=request-id] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 0, Skipped: 0, Total: 0","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_ValidatePreInstallChecksActivity/returns_error_when_validations_return_error[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_ValidatePreInstallChecksActivity/returns_error_when_validations_fail (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_ValidatePreInstallChecksActivity/returns_error_when_validations_return_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_ProvisionStorageActivity/returns_error_when_GetFilesystem_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_ProvisionStorageActivity/returns_error_when_ProvisionAsync_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_ProvisionStorageActivity/returns_error_when_GetFilesystem_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_ProvisionStorageActivity/returns_error_when_ProvisionAsync_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_ReadDomainsActivity/returns_error_when_GetEnvironment_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_ReadDomainsActivity/returns_error_when_GetDomainName_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_ReadDomainsActivity/returns_error_when_GetEnvironment_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_ReadDomainsActivity/returns_error_when_GetDomainName_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_SetCMLVersionActivity/return_error_when_SetCDSWVersion_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0159674,"caller":"core/install_activities.go:771","msg":"Failed to set CDSW version to '' for workspaceCrn 'workspaceCrn'. Error 'SetCDSWVersion error'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).SetCMLVersionActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:771
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_SetCMLVersionActivity.func3
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:2916
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_SetCMLVersionActivity/return_error_when_SetCDSWVersion_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_SetLonghornClusterIPActivity/returns_error_when_GetAdminKubeConfig_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_SetLonghornClusterIPActivity/returns_error_when_GetLonghornClusterIP_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_SetLonghornClusterIPActivity/returns_error_when_SetLonghornClusterIP_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_SetLonghornClusterIPActivity/returns_error_when_GetAdminKubeConfig_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_SetLonghornClusterIPActivity/returns_error_when_GetLonghornClusterIP_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_SetLonghornClusterIPActivity/returns_error_when_SetLonghornClusterIP_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/returns_error_for_Azure_with_no_NfsServer[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/returns_error_for_Azure_with_no_NfsVersion[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/returns_error_for_Azure_with_different_NfsServer_and_FileSystemId[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/returns_error_for_AWS_with_no_FileSystemId[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/Database_write_failure_causes_error[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/returns_error_for_Azure_with_no_NfsServer (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/returns_error_for_Azure_with_no_NfsVersion (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/returns_error_for_Azure_with_different_NfsServer_and_FileSystemId (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/returns_error_for_AWS_with_no_FileSystemId (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_UpdateStorageInDatabaseActivity/Database_write_failure_causes_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0380282,"caller":"core/install_activities.go:1037","msg":"Install Retry failed: 'unexpected error'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).ReinstallCMLWorkspaceActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1037
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_ReinstallCMLWorkspaceActivity.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3577
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/returns_success[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"info","ts":1706912468.0447254,"caller":"core/install_activities.go:1056","msg":"Running Install Failure Diagnostics for releaseName 'mlx-mlx' in namespace 'mlx' for workspaceCrn 'test-crn-ws'.","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.04564,"caller":"core/install_activities.go:1136","msg":"Persistent Volume is not bound: PVName=pv-2","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1136
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/fails_to_get_kubeconfig[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0481236,"caller":"core/install_activities.go:1052","msg":"Failed to get kubeconfig: kubeconfig error","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1052
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/fails_to_list_nodes[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"info","ts":1706912468.0501986,"caller":"core/install_activities.go:1056","msg":"Running Install Failure Diagnostics for releaseName 'mlx-mlx' in namespace 'mlx' for workspaceCrn 'test-crn-ws'.","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0504043,"caller":"core/install_activities.go:1060","msg":"Failed to list nodes: node list error","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1060
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/fails_to_list_pods[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"info","ts":1706912468.0525815,"caller":"core/install_activities.go:1056","msg":"Running Install Failure Diagnostics for releaseName 'mlx-mlx' in namespace 'mlx' for workspaceCrn 'test-crn-ws'.","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.053137,"caller":"core/install_activities.go:1106","msg":"Failed to list pods: pod list error","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1106
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/helm_release_in_bad_state[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"info","ts":1706912468.0551724,"caller":"core/install_activities.go:1056","msg":"Running Install Failure Diagnostics for releaseName 'mlx-mlx' in namespace 'mlx' for workspaceCrn 'test-crn-ws'.","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.05567,"caller":"core/install_activities.go:1097","msg":"Helm release for releaseName 'mlx-mlx' in namespace 'mlx' for workspaceCrn 'test-crn-ws' is in a bad state. Status: failed and Description: Release is in a bad state","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1097
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/node_not_ready[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"info","ts":1706912468.0579593,"caller":"core/install_activities.go:1056","msg":"Running Install Failure Diagnostics for releaseName 'mlx-mlx' in namespace 'mlx' for workspaceCrn 'test-crn-ws'.","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.058167,"caller":"core/install_activities.go:1074","msg":"ML infra node is not healthy: NodeName=node-3","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1074
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0583813,"caller":"core/install_activities.go:1081","msg":"ML infra node count is less than 2: Count=1","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1081
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/persistent_volume_not_bound[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"info","ts":1706912468.0607226,"caller":"core/install_activities.go:1056","msg":"Running Install Failure Diagnostics for releaseName 'mlx-mlx' in namespace 'mlx' for workspaceCrn 'test-crn-ws'.","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0615158,"caller":"core/install_activities.go:1136","msg":"Persistent Volume is not bound: PVName=pv-1","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1136
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0617342,"caller":"core/install_activities.go:1136","msg":"Persistent Volume is not bound: PVName=pv-2","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).RunInstallFailureDiagnosticActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1136
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity.func8
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:3925
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: --- PASS: TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity (0.02s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/returns_success (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/fails_to_get_kubeconfig (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/fails_to_list_nodes (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/fails_to_list_pods (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/helm_release_in_bad_state (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/node_not_ready (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_RunInstallFailureDiagnosticActivity/persistent_volume_not_bound (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity/returns_success_for_a_deployed_Helm_release[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity/returns_error_for_unreachable_Kubernetes_API_server[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0671349,"caller":"core/install_activities.go:1206","msg":"Unable to ping the Kubernetes API server: API server unreachable. Please check your network configuration to ensure that the CML control plane can communicate with the cluster. If the cluster is private, please follow the documentation at https://docs.cloudera.com/cdp-public-cloud-preview-features/cloud/ml-private-cluster-support/ml-private-cluster-support.pdf","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).FindHelmInstallFailureCauseActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1206
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity.func4
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:4063
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity/returns_error_for_Helm_release_in_a_bad_state[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.0697095,"caller":"core/install_activities.go:1221","msg":"Helm release in a bad state: releaseName: 'mlx-mlx', namespace: 'mlx', workspaceCrn: 'test-crn-ws', Status: failed, Description: Helm release failed","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallWorkspaceActivitiesV1).FindHelmInstallFailureCauseActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities.go:1221
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity.func4
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_activities_test.go:4063
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: --- PASS: TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity (0.01s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity/returns_success_for_a_deployed_Helm_release (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity/returns_error_for_unreachable_Kubernetes_API_server (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallWorkspaceActivitiesV1_FindHelmInstallFailureCauseActivity/returns_error_for_Helm_release_in_a_bad_state (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckAllContainersAreReadyActivity/getKubeConfig_throws_error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckAllContainersAreReadyActivity/GetAdminKubeConfig_throws_error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckAllContainersAreReadyActivity/List_pod_throws_error[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckAllContainersAreReadyActivity/getKubeConfig_throws_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckAllContainersAreReadyActivity/GetAdminKubeConfig_throws_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckAllContainersAreReadyActivity/List_pod_throws_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/GetAdminKubeConfig_throws_error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_unable_to_GetReleaseStatus_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus_is_nil[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info_is_nil[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info.Status_==_release.StatusFailed[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info.Status_==_release.StatusDeployed_and_GetReleaseStatus.Info.Description_!=_Install_complete[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/GetAdminKubeConfig_throws_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_unable_to_GetReleaseStatus_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus_is_nil (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info_is_nil (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info.Status_==_release.StatusFailed (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_CheckHelmReleaseStatusSuccessfulActivity/returns_error_when_GetReleaseStatus.Info.Status_==_release.StatusDeployed_and_GetReleaseStatus.Info.Description_!=_Install_complete (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_DeleteHelmReleaseActivity/returns_error_when_GetAdminKubeConfig_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_DeleteHelmReleaseActivity/returns_error_when_helm.DeleteUsingSDK_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_DeleteHelmReleaseActivity/returns_error_when_GetAdminKubeConfig_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_DeleteHelmReleaseActivity/returns_error_when_helm.DeleteUsingSDK_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_DoesHelmReleaseExistActivity/returns_error_when_GetAdminKubeConfig_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_DoesHelmReleaseExistActivity/returns_false_when_GetReleaseStatus_throws_error[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_DoesHelmReleaseExistActivity/returns_error_when_GetAdminKubeConfig_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_DoesHelmReleaseExistActivity/returns_false_when_GetReleaseStatus_throws_error (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_InstallModelRegistryActivity/returns_error_when_helm_install_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:324","message":"[requestId=request-id] Failed helm install: InstallAsync error
","requestId":"request-id","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:325","message":"[requestId=request-id] helm stdout: 
","requestId":"request-id","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:326","message":"[requestId=request-id] helm stderr: 
","requestId":"request-id","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.1443229,"caller":"core/install_resource_activities.go:505","msg":"Installation failed: 'InstallAsync error'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallResourceActivitiesV1).InstallModelRegistryActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_resource_activities.go:505
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallModelRegistryActivities_InstallModelRegistryActivity.func4
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_resource_activities_test.go:1090
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_InstallModelRegistryActivity/returns_error_when_GetAdminKubeConfig_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:307","message":"[requestId=request-id] Failed to get kubeconfig: GetAdminKubeConfig error","requestId":"request-id","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.1468103,"caller":"core/install_resource_activities.go:505","msg":"Installation failed: 'Failed to get kubeconfig: GetAdminKubeConfig error'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallResourceActivitiesV1).InstallModelRegistryActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_resource_activities.go:505
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallModelRegistryActivities_InstallModelRegistryActivity.func4
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_resource_activities_test.go:1090
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_InstallModelRegistryActivity/returns_error_when_helm_install_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_InstallModelRegistryActivity/returns_error_when_GetAdminKubeConfig_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_PatchDnsInPublicCloudActivity/returns_error_when_GetAdminKubeConfig_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_PatchDnsInPublicCloudActivity/returns_error_when_GetCloudSession_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_PatchDnsInPublicCloudActivity/returns_error_when_RegisterDnsV2_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"info","line":"core/core_install.go:388","message":"[requestId=request-id] Failed to register dns for : RegisterDnsV2 error
","requestId":"request-id","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_PatchDnsInPublicCloudActivity/returns_error_when_CreateFreeIpaDnsEntry_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/core_install.go:393","message":"[requestId=request-id] failed to create DNS entry in FreeIpa: CreateFreeIpaDnsEntry error","requestId":"request-id","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_PatchDnsInPublicCloudActivity/returns_error_when_GetAdminKubeConfig_fails (0.01s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_PatchDnsInPublicCloudActivity/returns_error_when_GetCloudSession_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_PatchDnsInPublicCloudActivity/returns_error_when_RegisterDnsV2_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_PatchDnsInPublicCloudActivity/returns_error_when_CreateFreeIpaDnsEntry_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_PostInstallActivity/returns_error_when_GetAdminKubeConfig_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_PostInstallActivity/returns_error_when_PostInstallV2_fails[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"info","ts":1706912468.171434,"caller":"core/install_resource_activities.go:97","msg":"Failed to complete post install operations: PostInstallV2 error
","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id"}[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_PostInstallActivity/returns_error_when_GetAdminKubeConfig_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_PostInstallActivity/returns_error_when_PostInstallV2_fails (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_ReadDomainsActivity/returns_error_when_GetEnvironment_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_ReadDomainsActivity/returns_error_when_GetDomainName_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_ReadDomainsActivity/returns_error_when_GetEnvironment_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_ReadDomainsActivity/returns_error_when_GetDomainName_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestInstallModelRegistryActivities_SetResourceVersionActivity/return_error_when_SetCDSWVersion_errors[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.1763115,"caller":"core/install_resource_activities.go:142","msg":"Failed to set resource version to '' for resourceCrn 'resourceCrn'. Error 'SetCDSWVersion error'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*InstallResourceActivitiesV1).SetResourceVersionActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_resource_activities.go:142
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestInstallModelRegistryActivities_SetResourceVersionActivity.func3
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/install_resource_activities_test.go:1552
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestInstallModelRegistryActivities_SetResourceVersionActivity/return_error_when_SetCDSWVersion_errors (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestUnitTestSuite/Test_InstallCMLChartWorkflow_with_existing_helm_install_failure[0m
2024-02-02 22:22:15 - [31mERROR-root::util|431:: Workspace Installation failed with error: unexpected error. Upon running diagnostics, the following error was found: unexpected error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestUnitTestSuite/Test_InstallCMLChartWorkflow_with_existing_install_failure_with_no_helm_install_failure[0m
2024-02-02 22:22:15 - INFO-root::util|431:: unexpected error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestUnitTestSuite/Test_InstallWorkflow_error_updates_workspace_status_change_to_failed[0m
2024-02-02 22:22:15 - INFO-root::util|431:: unexpected error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: Activity for launching pre-install validation pod failed: unexpected error[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestUnitTestSuite/Test_InstallCMLChartWorkflow_with_existing_helm_install_failure (0.01s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestUnitTestSuite/Test_InstallCMLChartWorkflow_with_existing_install_failure_with_no_helm_install_failure (0.01s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestUnitTestSuite/Test_InstallWorkflow_error_updates_workspace_status_change_to_failed (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.2913055,"caller":"core/modify_workflow.go:106","msg":"Instance group of name mlcpu1 is not present in the cluster.","requestId":"test-request-id","actorCrn":"test-actor-crn","resourceId":"cool-workspace-name","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*ModifyClusterInstanceGroupWorkflow).RunWorkflow
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/modify_workflow.go:106
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestModifyInstanceWorkflow.func2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/modify_workflow_test.go:191
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.3486905,"caller":"core/modify_workflow.go:562","msg":"Instance group of name mlcpu1 is not present in the cluster.","requestId":"test-request-id","actorCrn":"test-actor-crn","resourceId":"cool-workspace-name","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*DeleteClusterInstanceGroupWorkflow).RunWorkflow
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/modify_workflow.go:562
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestDeleteInstanceWorkflow.func2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/modify_workflow_test.go:825
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_AreAllMountTargetsReadyActivity_err_causes_workflow_failure[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.399Z	ERROR	core/provision_storage_workflow.go:198	AreAllMountTargetsReadyActivity failed with error 'unexpected error'	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: Mount targets  failed to become ready for access point '' for file system ''[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_InstallCSIDriversActivity_error_causes_workflow_failure[0m
2024-02-02 22:22:15 - INFO-root::util|431:: Activity for installing CSI drivers failed: unexpected error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_IsFileSystemReadyActivity_error_causes_retry[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.450Z	ERROR	core/provision_storage_workflow.go:125	IsFileSystemReadyActivity failed with error 'unexpected error'	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_UpdateStorageInDatabaseActivity_error_causes_workflow_failure[0m
2024-02-02 22:22:15 - INFO-root::util|431:: Activity for updating the storage in database failed: unexpected error[0m
2024-02-02 22:22:15 - INFO-root::util|431:: === RUN   TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_error_updates_workspace_status_change_to_failed[0m
2024-02-02 22:22:15 - INFO-root::util|431:: unexpected error[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_AreAllMountTargetsReadyActivity_err_causes_workflow_failure (0.01s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_InstallCSIDriversActivity_error_causes_workflow_failure (0.01s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_IsFileSystemReadyActivity_error_causes_retry (0.02s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_UpdateStorageInDatabaseActivity_error_causes_workflow_failure (0.01s)[0m
2024-02-02 22:22:15 - INFO-root::util|431::     --- PASS: TestProvisionStorageUnitTestSuite/Test_InstallWorkflow_error_updates_workspace_status_change_to_failed (0.00s)[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.4868,"caller":"core/resource_activities.go:73","msg":"Unable to fetch block volume info of for resourceCrn 'crn:cdp:ml:us-west-1:xxxTentacion:model_registry:85a26de4-36de-4f1b-8e9a-ab14b5ea9809' in namespace mlx due to error : unexpected error","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*ResourceActivitiesImpl).DeleteBlockVolumesUsedByResourceActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities.go:73
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestModelRegistryActivities.func1.2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities_test.go:106
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.4918065,"caller":"core/resource_activities.go:114","msg":"Unable to fetch persistent volumes for resourceCrn 'crn:cdp:ml:us-west-1:xxxTentacion:model_registry:85a26de4-36de-4f1b-8e9a-ab14b5ea9809' in namespace mlx due to error : unexpected error","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*ResourceActivitiesImpl).DeletePersistentVolumesUsedByResourceActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities.go:114
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestModelRegistryActivities.func2.2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities_test.go:189
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.496844,"caller":"core/resource_activities.go:157","msg":"Unable to fetch persistent volume claims for resourceCrn 'crn:cdp:ml:us-west-1:xxxTentacion:model_registry:85a26de4-36de-4f1b-8e9a-ab14b5ea9809' in namespace mlx due to error : unexpected error","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*ResourceActivitiesImpl).DeletePersistentVolumeClaimsUsedByResourceActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities.go:157
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestModelRegistryActivities.func3.2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities_test.go:276
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.5015852,"caller":"core/resource_activities.go:314","msg":"Failed to fetch the pods from namespace mlx : %!(EXTRA *errors.errorString=unexpected error)","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*ResourceActivitiesImpl).RemovePodsActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities.go:314
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestModelRegistryActivities.func4.2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities_test.go:349
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.5060058,"caller":"core/resource_activities.go:268","msg":"Failed to fetch the jobs from namespace mlx : %!(EXTRA *errors.errorString=unexpected error)","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*ResourceActivitiesImpl).RemoveJobsActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities.go:268
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestModelRegistryActivities.func5.2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities_test.go:421
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: {"level":"error","ts":1706912468.5105295,"caller":"core/resource_activities.go:222","msg":"Unable to fetch secrets for resourceCrn 'crn:cdp:ml:us-west-1:xxxTentacion:model_registry:85a26de4-36de-4f1b-8e9a-ab14b5ea9809' in namespace mlx due to error : unexpected error","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*ResourceActivitiesImpl).DeleteSecretsActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities.go:222
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestModelRegistryActivities.func6.2
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/resource_activities_test.go:491
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: Activity for mounting the storage failed[0m
2024-02-02 22:22:15 - INFO-root::util|431:: Block volumes storage failed to become ready[0m
2024-02-02 22:22:15 - INFO-root::util|431:: Activity for mounting the storage failed[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.581Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.581Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.581Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.581Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.581Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.582Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.582Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.582Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:15 - INFO-root::util|431:: 2024-02-02T22:21:08.582Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024-02-02T22:21:08.582Z	ERROR	core/restore_storage_workflow.go:179	IsAccessPointReadyActivity failed with error: unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: The access point '' for the file system '' failed to become ready[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Activity for fetching the restored block volumes failed: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Activity for fetching the restored NFS failed: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Activity for restoring the storage failed: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Activity for fetching the restored block volumes failed: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Activity for fetching the restored NFS failed: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Activity for updating the storage in the database failed: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: The file system '' failed to become ready[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestRestoreStorageUnitTestSuite/Test_RestoreWorkflow_error_updates_workspace_status_change_to_failed[0m
2024-02-02 22:22:16 - INFO-root::util|431:: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestRestoreStorageUnitTestSuite/Test_RestoreWorkflow_error_updates_workspace_status_change_to_failed (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431:: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024-02-02T22:21:08.695Z	ERROR	core/restore_workflow.go:134	Failed to TrackWorkspaceStatusChange 'provision:failed' due to error 'unexpected error'	{"requestId": "request-id", "actorCrn": "crn:altus:iam:us-west-1:9d74eee4-7ccf9edbb73d:user:a325601f-d52a-4cd49fbesdas", "resourceId": "ut-1"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to create Computex cluster: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to create Computex cluster: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024-02-02T22:21:08.708Z	ERROR	core/restore_workflow.go:329	Failed to TrackWorkspaceStatusChange 'installation:finished' due to error 'unexpected error'	{"requestId": "request-id", "actorCrn": "crn:altus:iam:us-west-1:9d74eee4-7ccf9edbb73d:user:a325601f-d52a-4cd49fbesdas", "resourceId": "ut-1"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to track the workspace status changes: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to install workspace: unexpected error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestStorageWorkspaceActivitiesV1_StartBlockVolumesRestoreActivity/error_during_block_volumes_restore[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestStorageWorkspaceActivitiesV1_StartBlockVolumesRestoreActivity/error_during_block_volumes_restore (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestStorageWorkspaceActivitiesV1_StartNFSRestoreActivity/error_during_NFS_restore[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestStorageWorkspaceActivitiesV1_StartNFSRestoreActivity/error_during_NFS_restore (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestStorageWorkspaceActivitiesV1_AreRestoreJobCompletedActivity/error_during_restore_job_check[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestStorageWorkspaceActivitiesV1_AreRestoreJobCompletedActivity/error_during_restore_job_check (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestStorageWorkspaceActivitiesV1_ListRestoreVolumeInfoActivity/error_during_list_restore_volume_info[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestStorageWorkspaceActivitiesV1_ListRestoreVolumeInfoActivity/error_during_list_restore_volume_info (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestStorageWorkspaceActivitiesV1_TagFileSystemUsedByWorkspaceActivity/error_during_tagging_of_file_system[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Failed to read required context from propagated context[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestStorageWorkspaceActivitiesV1_TagFileSystemUsedByWorkspaceActivity/error_during_tagging_of_file_system (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestPostProcessAfterRestoringNFS/pod_has_been_running_but_failed_to_be_deleted[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/storage_activities.go:420","message":"[requestId=request-id] Failed to delete the pre-install pod 'efs-cleanup-pod'. Error 'failed to delete pod'","requestId":"request-id","timestamp":"2024-02-02T22:21:18Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestPostProcessAfterRestoringNFS/create_pod_throws_error[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/storage_activities.go:657","message":"[requestId=request-id] Pre-install check pod failed to install with error 'failed to create pod'","requestId":"request-id","timestamp":"2024-02-02T22:21:18Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"core/storage_activities.go:472","message":"[requestId=request-id] Failed to delete the pre-install persistent volume 'efs-post-restore'. Error 'failed to delete pv'","requestId":"request-id","timestamp":"2024-02-02T22:21:18Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestPostProcessAfterRestoringNFS/pod_has_been_running_but_failed_to_be_deleted (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestPostProcessAfterRestoringNFS/create_pod_throws_error (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCleanUpPostProcessAfterRestore/failed_to_delete_pod[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"level":"error","ts":1706912478.764686,"caller":"core/storage_activities.go:254","msg":"Failed to delete existing pod: failed to delete pod
","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*StorageWorkspaceActivitiesV1).CleanupPostProcessAfterRestore
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/storage_activities.go:254
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestCleanUpPostProcessAfterRestore.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/storage_activities_test.go:999
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCleanUpPostProcessAfterRestore/failed_to_delete_pv[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"level":"error","ts":1706912478.7669766,"caller":"core/storage_activities.go:266","msg":"Failed to delete the pre-install persistent volume 'efs-post-restore-claim'. Error 'failed to delete pv'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*StorageWorkspaceActivitiesV1).CleanupPostProcessAfterRestore
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/storage_activities.go:266
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestCleanUpPostProcessAfterRestore.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/storage_activities_test.go:999
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCleanUpPostProcessAfterRestore/failed_to_delete_pvc[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"level":"error","ts":1706912478.7691545,"caller":"core/storage_activities.go:260","msg":"Failed to delete the pre-install persistent volume claim 'efs-post-restore-claim'. Error 'failed to delete pv'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.(*StorageWorkspaceActivitiesV1).CleanupPostProcessAfterRestore
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/storage_activities.go:260
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core.TestCleanUpPostProcessAfterRestore.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/storage_activities_test.go:999
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestCleanUpPostProcessAfterRestore/failed_to_delete_pod (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestCleanUpPostProcessAfterRestore/failed_to_delete_pv (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431::     --- PASS: TestCleanUpPostProcessAfterRestore/failed_to_delete_pvc (0.00s)[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Got Error: <nil>[0m
2024-02-02 22:22:16 - INFO-root::util|431:: Got Error: <nil>[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   Test_SuspendWorkflow_For_SuspendFailed_Workspace[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"level":"warn","ts":1706912478.7775853,"caller":"core/suspend_workflow.go:109","msg":"The workspace is in suspend:failed state, the original auto-scaling info will be used : %!(EXTRA map[string]models.AutoScalingGroup=map[])","requestId":"request-id","actorCrn":"actor-crn","resourceId":"cool-workspace-name"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"level":"warn","ts":1706912478.7777607,"caller":"core/suspend_workflow.go:139","msg":"The workspace is in suspend:failed state, the original replica info will be used : %!(EXTRA map[string]models.ReplicaInfo=map[])","requestId":"request-id","actorCrn":"actor-crn","resourceId":"cool-workspace-name"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: --- PASS: Test_SuspendWorkflow_For_SuspendFailed_Workspace (0.01s)[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=c924c646-c280-42ef-9860-c965ca83ca08] could not obtain default workload version from configuration variables in DB","requestId":"c924c646-c280-42ef-9860-c965ca83ca08","timestamp":"2024-02-02T22:21:21Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCreate/Preflight_failure[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === PAUSE TestCreate/Preflight_failure[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:3281","message":"[requestId=26057607-85dc-4b72-b8c8-a269de7abe4f] Existing workspace [test-workspace] found in namespace test-workspace-ns. Please remove the existing workspace, or specify a different namespace which does not have an existing workspace.","requestId":"26057607-85dc-4b72-b8c8-a269de7abe4f","timestamp":"2024-02-02T22:21:21Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024/02/02 22:21:21 Failed to start Create: Existing workspace [test-workspace] found in namespace test-workspace-ns. Please remove the existing workspace, or specify a different namespace which does not have an existing workspace.[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024/02/02 22:21:21 Failed to start Create: workspace named 'test workspace' already exists in environment 'test-environment'[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=c6a82d1e-43f6-43da-b213-6fdaac97c499] could not obtain default workload version from configuration variables in DB","requestId":"c6a82d1e-43f6-43da-b213-6fdaac97c499","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=463309da-1e18-4e62-bffb-80c584451089] could not obtain default workload version from configuration variables in DB","requestId":"463309da-1e18-4e62-bffb-80c584451089","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCreate/invalid_configurations_return_validation_failures[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=b53d673f-7c3d-4024-970e-514be7b49131] could not obtain default workload version from configuration variables in DB","requestId":"b53d673f-7c3d-4024-970e-514be7b49131","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=b53d673f-7c3d-4024-970e-514be7b49131] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 0, Skipped: 0, Total: 0","requestId":"b53d673f-7c3d-4024-970e-514be7b49131","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=b53d673f-7c3d-4024-970e-514be7b49131] {"id":"8848fd5d-a38f-40fb-951f-c93610071534","timestamp":1706912482713,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:d561fcbc-ceab-4a92-8789-9de20f8c2bac","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[],\"requestID\":\"b53d673f-7c3d-4024-970e-514be7b49131\"}","requestId":"b53d673f-7c3d-4024-970e-514be7b49131","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"b53d673f-7c3d-4024-970e-514be7b49131","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=b53d673f-7c3d-4024-970e-514be7b49131] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:d561fcbc-ceab-4a92-8789-9de20f8c2bac"}'","requestId":"b53d673f-7c3d-4024-970e-514be7b49131","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024/02/02 22:21:22 Failed to start Create: Pre-flight validations failed. {"failedValidations":[],"requestID":"b53d673f-7c3d-4024-970e-514be7b49131"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCreate/invalid_configuration_short_circuits_cluster_creation_and_returns_error[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=6e8f3c59-e796-4e90-850c-81485d0f7c51] could not obtain default workload version from configuration variables in DB","requestId":"6e8f3c59-e796-4e90-850c-81485d0f7c51","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=6e8f3c59-e796-4e90-850c-81485d0f7c51] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 0, Skipped: 0, Total: 0","requestId":"6e8f3c59-e796-4e90-850c-81485d0f7c51","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=6e8f3c59-e796-4e90-850c-81485d0f7c51] {"id":"1388190c-0e74-40e1-935d-2efd89845cc3","timestamp":1706912482920,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:20e80c1a-580f-4356-9164-3e3132534f62","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[],\"requestID\":\"6e8f3c59-e796-4e90-850c-81485d0f7c51\"}","requestId":"6e8f3c59-e796-4e90-850c-81485d0f7c51","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"6e8f3c59-e796-4e90-850c-81485d0f7c51","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=6e8f3c59-e796-4e90-850c-81485d0f7c51] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:20e80c1a-580f-4356-9164-3e3132534f62"}'","requestId":"6e8f3c59-e796-4e90-850c-81485d0f7c51","timestamp":"2024-02-02T22:21:22Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024/02/02 22:21:22 Failed to start Create: Pre-flight validations failed. {"failedValidations":[],"requestID":"6e8f3c59-e796-4e90-850c-81485d0f7c51"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCreate/invalid_configuration_with_skipValidation_short_circuits_cluster_creation_and_returns_error[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=1baf46e4-c796-44b8-b87c-a9c2fa31f439] could not obtain default workload version from configuration variables in DB","requestId":"1baf46e4-c796-44b8-b87c-a9c2fa31f439","timestamp":"2024-02-02T22:21:23Z"}[0m
2024-02-02 22:22:16 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=1baf46e4-c796-44b8-b87c-a9c2fa31f439] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 0, Skipped: 0, Total: 0","requestId":"1baf46e4-c796-44b8-b87c-a9c2fa31f439","timestamp":"2024-02-02T22:21:23Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=1baf46e4-c796-44b8-b87c-a9c2fa31f439] {"id":"1e211693-b3a2-460a-9904-2f3a8df8ef69","timestamp":1706912483095,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:a5a18922-d3ed-453e-9c6c-9ec950aa97f1","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[],\"requestID\":\"1baf46e4-c796-44b8-b87c-a9c2fa31f439\"}","requestId":"1baf46e4-c796-44b8-b87c-a9c2fa31f439","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"1baf46e4-c796-44b8-b87c-a9c2fa31f439","timestamp":"2024-02-02T22:21:23Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=1baf46e4-c796-44b8-b87c-a9c2fa31f439] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:a5a18922-d3ed-453e-9c6c-9ec950aa97f1"}'","requestId":"1baf46e4-c796-44b8-b87c-a9c2fa31f439","timestamp":"2024-02-02T22:21:23Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024/02/02 22:21:23 Failed to start Create: Pre-flight validations failed. {"failedValidations":[],"requestID":"1baf46e4-c796-44b8-b87c-a9c2fa31f439"}[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=b8a9fe53-6e15-4fca-82a0-8ab219982ede] could not obtain default workload version from configuration variables in DB","requestId":"b8a9fe53-6e15-4fca-82a0-8ab219982ede","timestamp":"2024-02-02T22:21:23Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024/02/02 22:21:23 Failed to start Create: Liftie pre-flight validation checks cannot be skipped[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=93b98d21-2fef-42cf-b276-9ee12c7d27f4] could not obtain default workload version from configuration variables in DB","requestId":"93b98d21-2fef-42cf-b276-9ee12c7d27f4","timestamp":"2024-02-02T22:21:23Z"}[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=557d94e4-0097-42f5-b331-c1eb5f3b6250] could not obtain default workload version from configuration variables in DB","requestId":"557d94e4-0097-42f5-b331-c1eb5f3b6250","timestamp":"2024-02-02T22:21:23Z"}[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=582344ab-17c5-4d0f-9826-258cfa43b298] could not obtain default workload version from configuration variables in DB","requestId":"582344ab-17c5-4d0f-9826-258cfa43b298","timestamp":"2024-02-02T22:21:24Z"}[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=2ae6ca00-b2fd-4ebc-8ca9-f3c323de7df4] could not obtain default workload version from configuration variables in DB","requestId":"2ae6ca00-b2fd-4ebc-8ca9-f3c323de7df4","timestamp":"2024-02-02T22:21:24Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCreate/failure_to_list_roles_for_user_throws_correct_error[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=9d80e130-2efa-47ea-9bdb-43d308e97eb2] could not obtain default workload version from configuration variables in DB","requestId":"9d80e130-2efa-47ea-9bdb-43d308e97eb2","timestamp":"2024-02-02T22:21:24Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024/02/02 22:21:24 Failed to start Create: Failed to ListRoles from UserManagementService for accountId 'testAccountID'. Err 'unable to list roles for account'.[0m
2024-02-02 22:22:16 - INFO-root::util|431:: === RUN   TestCreate/failure_on_spawning_a_cluster_with_empty_instance_groups[0m
2024-02-02 22:22:16 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=0aa335c8-ec6c-4ac9-ac9f-8227b577d395] could not obtain default workload version from configuration variables in DB","requestId":"0aa335c8-ec6c-4ac9-ac9f-8227b577d395","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:16 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=0aa335c8-ec6c-4ac9-ac9f-8227b577d395] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"0aa335c8-ec6c-4ac9-ac9f-8227b577d395","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=0aa335c8-ec6c-4ac9-ac9f-8227b577d395] {"id":"9c366611-1582-4214-9fd6-186022822769","timestamp":1706912485183,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:bcbe966f-6175-45a0-92f4-c03a753f0383","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[{\"category\":\"COMMON\",\"name\":\"Instance Groups basic parameter validation\",\"detailedMessage\":\"instance groups cannot be an empty list\"}],\"requestID\":\"0aa335c8-ec6c-4ac9-ac9f-8227b577d395\"}","requestId":"0aa335c8-ec6c-4ac9-ac9f-8227b577d395","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"0aa335c8-ec6c-4ac9-ac9f-8227b577d395","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=0aa335c8-ec6c-4ac9-ac9f-8227b577d395] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:bcbe966f-6175-45a0-92f4-c03a753f0383"}'","requestId":"0aa335c8-ec6c-4ac9-ac9f-8227b577d395","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:16 - INFO-root::util|431:: 2024/02/02 22:21:25 Failed to start Create: Pre-flight validations failed. {"failedValidations":[{"category":"COMMON","name":"Instance Groups basic parameter validation","detailedMessage":"instance groups cannot be an empty list"}],"requestID":"0aa335c8-ec6c-4ac9-ac9f-8227b577d395"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da] could not obtain default workload version from configuration variables in DB","requestId":"ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:17 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da] {"id":"1fbbbc68-54eb-42e5-b174-a18b5859b4b2","timestamp":1706912485464,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:86578211-cee9-41d9-b178-27e35df0b8f7","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[{\"category\":\"COMMON\",\"name\":\"Instance type region availability validation\",\"detailedMessage\":\"Instance type is not supported in the given region. This validation is non-skippable as workspaces with unsupported instance types cannot be created.\"}],\"requestID\":\"ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da\"}","requestId":"ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:86578211-cee9-41d9-b178-27e35df0b8f7"}'","requestId":"ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:25 Failed to start Create: Pre-flight validations failed. {"failedValidations":[{"category":"COMMON","name":"Instance type region availability validation","detailedMessage":"Instance type is not supported in the given region. This validation is non-skippable as workspaces with unsupported instance types cannot be created."}],"requestID":"ab41c2f6-83aa-487a-a4e5-e1bb2ce3c8da"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=41d9e6b7-ca56-492c-bc04-7925904f71b3] could not obtain default workload version from configuration variables in DB","requestId":"41d9e6b7-ca56-492c-bc04-7925904f71b3","timestamp":"2024-02-02T22:21:25Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === RUN   TestCreate/failure_on_spawning_a_cluster_with_instance_groups_with_max_instance_set_to_0[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=4c355724-80c8-4a69-9cc7-18971fb65d7a] could not obtain default workload version from configuration variables in DB","requestId":"4c355724-80c8-4a69-9cc7-18971fb65d7a","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=4c355724-80c8-4a69-9cc7-18971fb65d7a] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"4c355724-80c8-4a69-9cc7-18971fb65d7a","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=4c355724-80c8-4a69-9cc7-18971fb65d7a] {"id":"b0699180-e29e-4c42-9dcb-7bb997791028","timestamp":1706912486129,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:fc6f3491-b6b4-4384-b9cb-250efe22e539","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[{\"category\":\"COMMON\",\"name\":\"Instance Groups basic parameter validation\",\"detailedMessage\":\"There has to be atleast one instance group with max instances greater than 0\"}],\"requestID\":\"4c355724-80c8-4a69-9cc7-18971fb65d7a\"}","requestId":"4c355724-80c8-4a69-9cc7-18971fb65d7a","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"4c355724-80c8-4a69-9cc7-18971fb65d7a","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=4c355724-80c8-4a69-9cc7-18971fb65d7a] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:fc6f3491-b6b4-4384-b9cb-250efe22e539"}'","requestId":"4c355724-80c8-4a69-9cc7-18971fb65d7a","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 Failed to start Create: Pre-flight validations failed. {"failedValidations":[{"category":"COMMON","name":"Instance Groups basic parameter validation","detailedMessage":"There has to be atleast one instance group with max instances greater than 0"}],"requestID":"4c355724-80c8-4a69-9cc7-18971fb65d7a"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=a9e08acc-88fa-4346-af9e-17c23af38cce] could not obtain default workload version from configuration variables in DB","requestId":"a9e08acc-88fa-4346-af9e-17c23af38cce","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a9e08acc-88fa-4346-af9e-17c23af38cce] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a9e08acc-88fa-4346-af9e-17c23af38cce","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a9e08acc-88fa-4346-af9e-17c23af38cce] {"id":"384154aa-fbfc-4548-b3fa-dded3f0ea2c2","timestamp":1706912486161,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:7f7b04c4-e69d-4e68-94c4-bdae8a1cddba","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[{\"category\":\"COMMON\",\"name\":\"Instance Groups basic parameter validation\",\"detailedMessage\":\"MinInstances cannot be greater than MaxInstances in autoscaling groups\"}],\"requestID\":\"a9e08acc-88fa-4346-af9e-17c23af38cce\"}","requestId":"a9e08acc-88fa-4346-af9e-17c23af38cce","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"a9e08acc-88fa-4346-af9e-17c23af38cce","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=a9e08acc-88fa-4346-af9e-17c23af38cce] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:7f7b04c4-e69d-4e68-94c4-bdae8a1cddba"}'","requestId":"a9e08acc-88fa-4346-af9e-17c23af38cce","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 Failed to start Create: Pre-flight validations failed. {"failedValidations":[{"category":"COMMON","name":"Instance Groups basic parameter validation","detailedMessage":"MinInstances cannot be greater than MaxInstances in autoscaling groups"}],"requestID":"a9e08acc-88fa-4346-af9e-17c23af38cce"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === CONT  TestCreate/Preflight_failure[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=80846782-9ca7-431c-85f1-f96a540e56c6] could not obtain default workload version from configuration variables in DB","requestId":"80846782-9ca7-431c-85f1-f96a540e56c6","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=520273f1-e4e2-4b8e-baa4-13009235dd5f] could not obtain default workload version from configuration variables in DB","requestId":"520273f1-e4e2-4b8e-baa4-13009235dd5f","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:2772","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] Start  CreateWorkspace. Request is {ProvisionData:{InstanceGroups:[] Environment: Tags:[] Network:{Plugin: Options:{Encryption:} Topology:{Subnets:[]} ServiceCidr: InboundProxyCidr:[] PodCidr: OutboundType: Aws:<nil>} UseLegacyNodeLabel:false XEntitlements:[]} Environment:test-environment Namespace: WorkspaceName:testcreate workspace preflight failure Version: GBN: KubernetesVersion: Headers:{ActorCRN:crn:altus:environments:us-west-1:testAccountID:environment:testResourceID RequestID:30cc2a19-39bd-461a-a4d7-bc7d87fab993} DisableTLS:false UsePublicLoadBalancer:false DisableSSO:false EnableMonitoring:false EnableYunikorn:false EnableGovernance:false ExistingNFSServer: NFSVersion: NfsStorageClass: ExistingK8sKubeconfig: LoadBalancerIPWhitelists:[] EnableModelMetrics:false ExistingDatabaseReq:{ExistingDatabaseHost: ExistingDatabasePort: ExistingDatabaseName: ExistingDatabaseUser: ExistingDatabasePassword:} UseSpotInstancesForMLInfra:false UseSpotInstancesForLiftieInfra:false WhitelistAuthorizedIPRanges:false AuthorizedIPRanges:[] AdditionalRuntimeRepo: GovernancePrincipal: SkipValidation:false CustomerKMSKeyArn: ImageCatalog:{Crn: Name:} PerformCDSWMigration:false CDSWKubeconfig: PrivateCluster:false UseLegacyStorageDrivers:false SubnetsForLoadBalancer:[] StaticSubdomain: EntitlementsOverrides:[] CDSWMigrationTimeoutMinutes:0 CdswMigrationMode: OutboundTypeUDR:false ResourcePoolConfig:{ParentPoolName: WorkspaceQuota:{CPUQuota: MemoryQuota: GPUQuota:} InfraQuota:{CPUQuota: MemoryQuota: GPUQuota:}} NfsDiskSize:0 EnableUmbraIntegration:false UseDefaultResourcePool:false CustomImages:{CustomTGTGenImages:{TGTGeneratorImage: TGTLoaderImage:} CustomSDXImage:{SDXInitContainerImage:} CustomMLGovImage:{MLGovImage:}} Upgrade:false}","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] could not obtain default workload version from configuration variables in DB","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=841e4f30-ce7b-4b66-8f79-e2fa8029185d] could not obtain default workload version from configuration variables in DB","requestId":"841e4f30-ce7b-4b66-8f79-e2fa8029185d","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=2198e0bb-830f-424e-bdea-500537e1017d] could not obtain default workload version from configuration variables in DB","requestId":"2198e0bb-830f-424e-bdea-500537e1017d","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:560","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] The preflight check for FreeIPA failed: &{ <nil>   <nil> <nil> <nil> <nil> <nil> [] <nil> <nil> <nil> UNAVAILABLE   <nil>  <nil> }","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 Failed to start Create: This environment free IPA is not ready[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:2772","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] Start  CreateWorkspace. Request is {ProvisionData:{InstanceGroups:[] Environment: Tags:[] Network:{Plugin: Options:{Encryption:} Topology:{Subnets:[]} ServiceCidr: InboundProxyCidr:[] PodCidr: OutboundType: Aws:<nil>} UseLegacyNodeLabel:false XEntitlements:[]} Environment:test-environment Namespace: WorkspaceName:testcreate workspace preflight failure Version: GBN: KubernetesVersion: Headers:{ActorCRN:crn:altus:environments:us-west-1:testAccountID:environment:testResourceID RequestID:30cc2a19-39bd-461a-a4d7-bc7d87fab993} DisableTLS:false UsePublicLoadBalancer:false DisableSSO:false EnableMonitoring:false EnableYunikorn:false EnableGovernance:false ExistingNFSServer: NFSVersion: NfsStorageClass: ExistingK8sKubeconfig: LoadBalancerIPWhitelists:[] EnableModelMetrics:false ExistingDatabaseReq:{ExistingDatabaseHost: ExistingDatabasePort: ExistingDatabaseName: ExistingDatabaseUser: ExistingDatabasePassword:} UseSpotInstancesForMLInfra:false UseSpotInstancesForLiftieInfra:false WhitelistAuthorizedIPRanges:false AuthorizedIPRanges:[] AdditionalRuntimeRepo: GovernancePrincipal: SkipValidation:false CustomerKMSKeyArn: ImageCatalog:{Crn: Name:} PerformCDSWMigration:false CDSWKubeconfig: PrivateCluster:false UseLegacyStorageDrivers:false SubnetsForLoadBalancer:[] StaticSubdomain: EntitlementsOverrides:[] CDSWMigrationTimeoutMinutes:0 CdswMigrationMode: OutboundTypeUDR:false ResourcePoolConfig:{ParentPoolName: WorkspaceQuota:{CPUQuota: MemoryQuota: GPUQuota:} InfraQuota:{CPUQuota: MemoryQuota: GPUQuota:}} NfsDiskSize:0 EnableUmbraIntegration:false UseDefaultResourcePool:false CustomImages:{CustomTGTGenImages:{TGTGeneratorImage: TGTLoaderImage:} CustomSDXImage:{SDXInitContainerImage:} CustomMLGovImage:{MLGovImage:}} Upgrade:false}","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] could not obtain default workload version from configuration variables in DB","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:547 [35;1mUNIQUE constraint failed: audit_event.request_id[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 Failed to add audit event:  UNIQUE constraint failed: audit_event.request_id[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4247","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] UNIQUE constraint failed: audit_event.request_id","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:526","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] The preflight check for Cloudbreak failed:  UNAVAILABLE","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 Failed to start Create: This environment is not ready[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:2772","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] Start  CreateWorkspace. Request is {ProvisionData:{InstanceGroups:[] Environment: Tags:[] Network:{Plugin: Options:{Encryption:} Topology:{Subnets:[]} ServiceCidr: InboundProxyCidr:[] PodCidr: OutboundType: Aws:<nil>} UseLegacyNodeLabel:false XEntitlements:[]} Environment:test-environment Namespace: WorkspaceName:testcreate workspace preflight failure Version: GBN: KubernetesVersion: Headers:{ActorCRN:crn:altus:environments:us-west-1:testAccountID:environment:testResourceID RequestID:30cc2a19-39bd-461a-a4d7-bc7d87fab993} DisableTLS:false UsePublicLoadBalancer:false DisableSSO:false EnableMonitoring:false EnableYunikorn:false EnableGovernance:false ExistingNFSServer: NFSVersion: NfsStorageClass: ExistingK8sKubeconfig: LoadBalancerIPWhitelists:[] EnableModelMetrics:false ExistingDatabaseReq:{ExistingDatabaseHost: ExistingDatabasePort: ExistingDatabaseName: ExistingDatabaseUser: ExistingDatabasePassword:} UseSpotInstancesForMLInfra:false UseSpotInstancesForLiftieInfra:false WhitelistAuthorizedIPRanges:false AuthorizedIPRanges:[] AdditionalRuntimeRepo: GovernancePrincipal: SkipValidation:false CustomerKMSKeyArn: ImageCatalog:{Crn: Name:} PerformCDSWMigration:false CDSWKubeconfig: PrivateCluster:false UseLegacyStorageDrivers:false SubnetsForLoadBalancer:[] StaticSubdomain: EntitlementsOverrides:[] CDSWMigrationTimeoutMinutes:0 CdswMigrationMode: OutboundTypeUDR:false ResourcePoolConfig:{ParentPoolName: WorkspaceQuota:{CPUQuota: MemoryQuota: GPUQuota:} InfraQuota:{CPUQuota: MemoryQuota: GPUQuota:}} NfsDiskSize:0 EnableUmbraIntegration:false UseDefaultResourcePool:false CustomImages:{CustomTGTGenImages:{TGTGeneratorImage: TGTLoaderImage:} CustomSDXImage:{SDXInitContainerImage:} CustomMLGovImage:{MLGovImage:}} Upgrade:false}","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] could not obtain default workload version from configuration variables in DB","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=0822168b-3e9b-44e0-b5d7-0432043f2732] could not obtain default workload version from configuration variables in DB","requestId":"0822168b-3e9b-44e0-b5d7-0432043f2732","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=f6b755c8-7d9e-41cf-aea9-24eddc821062] could not obtain default workload version from configuration variables in DB","requestId":"f6b755c8-7d9e-41cf-aea9-24eddc821062","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:547 [35;1mUNIQUE constraint failed: audit_event.request_id[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 Failed to add audit event:  UNIQUE constraint failed: audit_event.request_id[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4247","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] UNIQUE constraint failed: audit_event.request_id","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:542","message":"[requestId=30cc2a19-39bd-461a-a4d7-bc7d87fab993] The preflight check for Datalake failed: &{    0   false false   <nil>  false    UNAVAILABLE  map[]}","requestId":"30cc2a19-39bd-461a-a4d7-bc7d87fab993","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:26 Failed to start Create: This environment datalake is not ready[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=63e73955-9d91-402a-918d-10f42d4097ab] could not obtain default workload version from configuration variables in DB","requestId":"63e73955-9d91-402a-918d-10f42d4097ab","timestamp":"2024-02-02T22:21:26Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431::     --- PASS: TestCreate/invalid_configurations_return_validation_failures (0.23s)[0m
2024-02-02 22:22:17 - INFO-root::util|431::     --- PASS: TestCreate/invalid_configuration_short_circuits_cluster_creation_and_returns_error (0.21s)[0m
2024-02-02 22:22:17 - INFO-root::util|431::     --- PASS: TestCreate/invalid_configuration_with_skipValidation_short_circuits_cluster_creation_and_returns_error (0.17s)[0m
2024-02-02 22:22:17 - INFO-root::util|431::     --- PASS: TestCreate/failure_to_list_roles_for_user_throws_correct_error (0.30s)[0m
2024-02-02 22:22:17 - INFO-root::util|431::     --- PASS: TestCreate/failure_on_spawning_a_cluster_with_empty_instance_groups (0.30s)[0m
2024-02-02 22:22:17 - INFO-root::util|431::     --- PASS: TestCreate/failure_on_spawning_a_cluster_with_instance_groups_with_max_instance_set_to_0 (0.34s)[0m
2024-02-02 22:22:17 - INFO-root::util|431::     --- PASS: TestCreate/Preflight_failure (0.78s)[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=7ffbb475-e0d6-462a-80b8-dba6558f6e75] could not obtain default workload version from configuration variables in DB","requestId":"7ffbb475-e0d6-462a-80b8-dba6558f6e75","timestamp":"2024-02-02T22:21:27Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=eca1f7da-8620-45a0-b92b-653d1dcdf3ff] could not obtain default workload version from configuration variables in DB","requestId":"eca1f7da-8620-45a0-b92b-653d1dcdf3ff","timestamp":"2024-02-02T22:21:27Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=06d89fbd-c2f3-4b4d-a9fa-1a8bfe164248] could not obtain default workload version from configuration variables in DB","requestId":"06d89fbd-c2f3-4b4d-a9fa-1a8bfe164248","timestamp":"2024-02-02T22:21:27Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=81b9cb90-ac94-47d0-b909-94012253dc8b] could not obtain default workload version from configuration variables in DB","requestId":"81b9cb90-ac94-47d0-b909-94012253dc8b","timestamp":"2024-02-02T22:21:27Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:27 Failed to start Create: Failed to start 'CreateWorkspaceWorkflow' for resource name 'testAccountID:test-environment:test workspace', workspace 'crn:cdp:ml:us-west-1:testAccountID:workspace:ebda66a2-413a-4b03-89de-d7d3a3fdfd19': previous workflow 'wf-1' is not closed[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === RUN   TestDelete/ClusterFailure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === PAUSE TestDelete/ClusterFailure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === RUN   TestDelete/StorageFailure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === PAUSE TestDelete/StorageFailure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === RUN   TestDelete/StorageFailureAndKeepStorage[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === PAUSE TestDelete/StorageFailureAndKeepStorage[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === RUN   TestDelete/Preflight_failure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === PAUSE TestDelete/Preflight_failure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === CONT  TestDelete/ClusterFailure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === CONT  TestDelete/StorageFailureAndKeepStorage[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === CONT  TestDelete/Preflight_failure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: === CONT  TestDelete/StorageFailure[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=1659f770-14dc-45be-808a-f679f95c723b] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"1659f770-14dc-45be-808a-f679f95c723b","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:5173","message":"[requestId=1659f770-14dc-45be-808a-f679f95c723b] Cannot get kubeconfig, failed to remove Route53 record. Please remove this manually on AWS","requestId":"1659f770-14dc-45be-808a-f679f95c723b","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:560","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] The preflight check for FreeIPA failed: &{ <nil>   <nil> <nil> <nil> <nil> <nil> [] <nil> <nil> <nil> UNAVAILABLE   <nil>  <nil> }","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6401","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Preflight check for deleteWorkspace failed. This environment free IPA is not ready","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6342","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Failed to delete workspace: This environment free IPA is not ready","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:118","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Failed to start Delete: This environment free IPA is not ready
","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:526","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] The preflight check for Cloudbreak failed:  UNAVAILABLE","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6401","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Preflight check for deleteWorkspace failed. This environment is not ready","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6342","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Failed to delete workspace: This environment is not ready","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:118","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Failed to start Delete: This environment is not ready
","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:542","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] The preflight check for Datalake failed: &{    0   false false   <nil>  false    UNAVAILABLE  map[]}","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6401","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Preflight check for deleteWorkspace failed. This environment datalake is not ready","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6342","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Failed to delete workspace: This environment datalake is not ready","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:118","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Failed to start Delete: This environment datalake is not ready
","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:542","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] The preflight check for Datalake failed: &{    0   false false   <nil>  false    UNAVAILABLE  map[]}","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6401","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Preflight check for deleteWorkspace failed. This environment datalake is not ready","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:6405","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Force delete requested, ignoring error","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:5173","message":"[requestId=20c08cb1-38a8-406f-90fd-f420a589afe2] Cannot get kubeconfig, failed to remove Route53 record. Please remove this manually on AWS","requestId":"20c08cb1-38a8-406f-90fd-f420a589afe2","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"DEPROVISION_STARTED","OldStatus":"PROVISION_FAILED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9"}'","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=320d1cd1-2592-483d-b7fd-9f9e2138eefd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"320d1cd1-2592-483d-b7fd-9f9e2138eefd","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=20c08cb1-38a8-406f-90fd-f420a589afe2] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"20c08cb1-38a8-406f-90fd-f420a589afe2","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=20c08cb1-38a8-406f-90fd-f420a589afe2] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"DEPROVISION_STARTED","OldStatus":"PROVISION_FAILED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9"}'","requestId":"20c08cb1-38a8-406f-90fd-f420a589afe2","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=320d1cd1-2592-483d-b7fd-9f9e2138eefd] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"DEPROVISION_STARTED","OldStatus":"PROVISION_FAILED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9"}'","requestId":"320d1cd1-2592-483d-b7fd-9f9e2138eefd","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:5173","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Cannot get kubeconfig, failed to remove Route53 record. Please remove this manually on AWS","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:5205","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Failed to wait for provisioning, but ignoring error. Error is Broken provisioner does nothing","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:17 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:17 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=d0bc2423-0667-4a38-a300-eca9cd708753] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6498","message":"[requestId=d0bc2423-0667-4a38-a300-eca9cd708753] Workspace deprovision error handler is called","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6499","message":"[requestId=d0bc2423-0667-4a38-a300-eca9cd708753] Delete failed: failed to delete TGTGenerator host keytab: Failed!
","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5241","message":"[requestId=20c08cb1-38a8-406f-90fd-f420a589afe2] Storage unmount error handler is called","requestId":"20c08cb1-38a8-406f-90fd-f420a589afe2","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5242","message":"[requestId=20c08cb1-38a8-406f-90fd-f420a589afe2] Storage unmounting failed: Broken provisioner does nothing
","requestId":"20c08cb1-38a8-406f-90fd-f420a589afe2","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=d0bc2423-0667-4a38-a300-eca9cd708753] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=d0bc2423-0667-4a38-a300-eca9cd708753] {"id":"39b03ec7-dec4-45c9-adcd-23a1baab18da","timestamp":1706912488256,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9","oldStatus":7,"newStatus":9,"failureReason":"failed to delete TGTGenerator host keytab: Failed!","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","liftieId":"liftie-abcd","userTags":"{}","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=d0bc2423-0667-4a38-a300-eca9cd708753] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"DEPROVISION_FAILED","OldStatus":"DEPROVISION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9"}'","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6358","message":"[requestId=d0bc2423-0667-4a38-a300-eca9cd708753] Failed to delete: failed to delete TGTGenerator host keytab: Failed!","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:123","message":"[requestId=d0bc2423-0667-4a38-a300-eca9cd708753] Failed to execute Delete: failed to delete TGTGenerator host keytab: Failed!
","requestId":"d0bc2423-0667-4a38-a300-eca9cd708753","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"DEPROVISION_STARTED","OldStatus":"PROVISION_FAILED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9"}'","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5241","message":"[requestId=320d1cd1-2592-483d-b7fd-9f9e2138eefd] Storage unmount error handler is called","requestId":"320d1cd1-2592-483d-b7fd-9f9e2138eefd","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5242","message":"[requestId=320d1cd1-2592-483d-b7fd-9f9e2138eefd] Storage unmounting failed: Broken provisioner does nothing
","requestId":"320d1cd1-2592-483d-b7fd-9f9e2138eefd","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:5258","message":"[requestId=320d1cd1-2592-483d-b7fd-9f9e2138eefd] Storage unmount failed, but ignoring error. Error is Broken provisioner does nothing","requestId":"320d1cd1-2592-483d-b7fd-9f9e2138eefd","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=42f03679-f08f-4b33-978f-e5c0f2a0d80f] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"42f03679-f08f-4b33-978f-e5c0f2a0d80f","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=42f03679-f08f-4b33-978f-e5c0f2a0d80f] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"DEPROVISION_STARTED","OldStatus":"DEPROVISION_FAILED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9"}'","requestId":"42f03679-f08f-4b33-978f-e5c0f2a0d80f","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:5193","message":"[requestId=42f03679-f08f-4b33-978f-e5c0f2a0d80f] Failed to delete tgtgen, but ignoring error. Error is Failed!","requestId":"42f03679-f08f-4b33-978f-e5c0f2a0d80f","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5267","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Cluster deprovision error handler is called","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5268","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Deprovision failed: Broken provisioner does nothing
","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6498","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Workspace deprovision error handler is called","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6499","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Delete failed: Broken provisioner does nothing
","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5241","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Storage unmount error handler is called","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5242","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Storage unmounting failed: Broken provisioner does nothing
","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] {"id":"db7d9fde-aed3-4f1f-96a2-5819074191c2","timestamp":1706912488357,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9","oldStatus":7,"newStatus":9,"failureReason":"Broken provisioner does nothing","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","liftieId":"liftie-abcd","userTags":"{}","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"DEPROVISION_FAILED","OldStatus":"DEPROVISION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9"}'","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=20c08cb1-38a8-406f-90fd-f420a589afe2] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"20c08cb1-38a8-406f-90fd-f420a589afe2","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:5258","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Storage unmount failed, but ignoring error. Error is Broken provisioner does nothing","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6358","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Failed to delete: Broken provisioner does nothing","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:123","message":"[requestId=4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686] Failed to execute Delete: Broken provisioner does nothing
","requestId":"4e1e4c47-8daf-45e8-a5ba-0ff3d16c5686","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=320d1cd1-2592-483d-b7fd-9f9e2138eefd] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"320d1cd1-2592-483d-b7fd-9f9e2138eefd","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=1659f770-14dc-45be-808a-f679f95c723b] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"1659f770-14dc-45be-808a-f679f95c723b","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=d1e7dc17-8591-49ad-bbf0-a44bda66b11c] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"d1e7dc17-8591-49ad-bbf0-a44bda66b11c","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5297","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Storage deprovision error handler is called","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5298","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Storage deprovision failed: Broken provisioner does nothing
","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6498","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Workspace deprovision error handler is called","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6499","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Delete failed: Broken provisioner does nothing
","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] {"id":"6df562c3-633b-4327-9a65-537e078856cf","timestamp":1706912488504,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9","oldStatus":7,"newStatus":9,"failureReason":"Broken provisioner does nothing","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","liftieId":"liftie-abcd","userTags":"{}","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"DEPROVISION_FAILED","OldStatus":"DEPROVISION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:f977fbed-e251-4c98-98ff-e11d269161b9"}'","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6358","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Failed to delete: Broken provisioner does nothing","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:123","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Failed to execute Delete: Broken provisioner does nothing
","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:28 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=42f03679-f08f-4b33-978f-e5c0f2a0d80f] Error fetching cluster tags for usage tracking.  Liftie Id: liftie-abcd  Continuing... Get "/liftie/api/v1/cluster/liftie-abcd": unsupported protocol scheme ""","requestId":"42f03679-f08f-4b33-978f-e5c0f2a0d80f","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431::     --- PASS: TestDelete/ClusterFailure (0.95s)[0m
2024-02-02 22:22:18 - INFO-root::util|431::     --- PASS: TestDelete/StorageFailureAndKeepStorage (0.98s)[0m
2024-02-02 22:22:18 - INFO-root::util|431::     --- PASS: TestDelete/Preflight_failure (1.04s)[0m
2024-02-02 22:22:18 - INFO-root::util|431::     --- PASS: TestDelete/StorageFailure (1.08s)[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=bc462fb2-7929-4edb-92ec-1643bb6c2492] cannot determine cloud provider ","requestId":"bc462fb2-7929-4edb-92ec-1643bb6c2492","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0793b00c-43ac-4873-b0db-01234e03f0de] cannot determine cloud provider ","requestId":"0793b00c-43ac-4873-b0db-01234e03f0de","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d5566d91-55a1-477c-a151-83ec55df1299] cannot determine cloud provider ","requestId":"d5566d91-55a1-477c-a151-83ec55df1299","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8d22946e-2263-4da6-9fba-af4f7cb41d2b] cannot determine cloud provider ","requestId":"8d22946e-2263-4da6-9fba-af4f7cb41d2b","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8d22946e-2263-4da6-9fba-af4f7cb41d2b] cannot determine cloud provider ","requestId":"8d22946e-2263-4da6-9fba-af4f7cb41d2b","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8d22946e-2263-4da6-9fba-af4f7cb41d2b] cannot determine cloud provider ","requestId":"8d22946e-2263-4da6-9fba-af4f7cb41d2b","timestamp":"2024-02-02T22:21:28Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=156db24a-8166-4c98-a500-999cf4b2437e] cannot determine cloud provider ","requestId":"156db24a-8166-4c98-a500-999cf4b2437e","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=5acdbc5b-bbe4-47e1-8a6c-35b4cf4008a8] cannot determine cloud provider ","requestId":"5acdbc5b-bbe4-47e1-8a6c-35b4cf4008a8","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=2617ca02-e362-4a4e-b1f0-9a34523250f7] cannot determine cloud provider ","requestId":"2617ca02-e362-4a4e-b1f0-9a34523250f7","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=1121f8ed-6617-4074-b0c0-d32abd2384e1] cannot determine cloud provider ","requestId":"1121f8ed-6617-4074-b0c0-d32abd2384e1","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4683","message":"[requestId=96aab78a-bd76-4f94-8c11-8a9ae0e68eea] random error","requestId":"96aab78a-bd76-4f94-8c11-8a9ae0e68eea","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:134","message":"[requestId=96aab78a-bd76-4f94-8c11-8a9ae0e68eea] Failed to List: random error
","requestId":"96aab78a-bd76-4f94-8c11-8a9ae0e68eea","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === RUN   TestList/TestListBackups/db_errors[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === RUN   TestList/TestListBackups/db_errors/error_in_db_call_#1[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5899","message":"[requestId=39482078-ef4b-4403-85d4-724509da5fa2] random db error ","requestId":"39482078-ef4b-4403-85d4-724509da5fa2","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === RUN   TestList/TestListBackups/db_errors/error_in_db_call_#2[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5904","message":"[requestId=39482078-ef4b-4403-85d4-724509da5fa2] random db error ","requestId":"39482078-ef4b-4403-85d4-724509da5fa2","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === RUN   TestList/TestListBackups/db_errors/error_in_db_call_#3[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5914","message":"[requestId=39482078-ef4b-4403-85d4-724509da5fa2] random db error ","requestId":"39482078-ef4b-4403-85d4-724509da5fa2","timestamp":"2024-02-02T22:21:29Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431::         --- PASS: TestList/TestListBackups/db_errors (0.00s)[0m
2024-02-02 22:22:18 - INFO-root::util|431::             --- PASS: TestList/TestListBackups/db_errors/error_in_db_call_#1 (0.00s)[0m
2024-02-02 22:22:18 - INFO-root::util|431::             --- PASS: TestList/TestListBackups/db_errors/error_in_db_call_#2 (0.00s)[0m
2024-02-02 22:22:18 - INFO-root::util|431::             --- PASS: TestList/TestListBackups/db_errors/error_in_db_call_#3 (0.00s)[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID2:environment:testResourceID2","level":"error","line":"core/core.go:6332","message":"[requestId=73d70a67-b117-4267-88f6-f385d2c8dad0] workspace not found:test workspace","requestId":"73d70a67-b117-4267-88f6-f385d2c8dad0","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID2:environment:testResourceID2","level":"error","line":"utils/utils.go:118","message":"[requestId=73d70a67-b117-4267-88f6-f385d2c8dad0] Failed to start Delete: workspace not found:test workspace
","requestId":"73d70a67-b117-4267-88f6-f385d2c8dad0","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:30 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID3","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=b8a3c82c-3e46-460a-b842-02371a7fa468] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"b8a3c82c-3e46-460a-b842-02371a7fa468","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID2:environment:testResourceID2","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=2f490899-2c01-4ae8-8f22-4f8c1d9367d2] could not obtain default workload version from configuration variables in DB","requestId":"2f490899-2c01-4ae8-8f22-4f8c1d9367d2","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID2:environment:testResourceID2","level":"error","line":"core/core.go:4886","message":"[requestId=2f490899-2c01-4ae8-8f22-4f8c1d9367d2] workspace not found:test workspace","requestId":"2f490899-2c01-4ae8-8f22-4f8c1d9367d2","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:30 Failed to start Upgrade: workspace not found:test workspace[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID3","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=e3d075e5-e889-47f8-81de-36aa66d09892] could not obtain default workload version from configuration variables in DB","requestId":"e3d075e5-e889-47f8-81de-36aa66d09892","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:30 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID3","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=e3d075e5-e889-47f8-81de-36aa66d09892] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"e3d075e5-e889-47f8-81de-36aa66d09892","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:30 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID3","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=b8a3c82c-3e46-460a-b842-02371a7fa468] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"b8a3c82c-3e46-460a-b842-02371a7fa468","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:30 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID3","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=e3d075e5-e889-47f8-81de-36aa66d09892] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"e3d075e5-e889-47f8-81de-36aa66d09892","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=3be3470e-9374-49f6-b64d-7aefeedab22c] could not obtain default workload version from configuration variables in DB","requestId":"3be3470e-9374-49f6-b64d-7aefeedab22c","timestamp":"2024-02-02T22:21:30Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:31 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3be3470e-9374-49f6-b64d-7aefeedab22c] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"3be3470e-9374-49f6-b64d-7aefeedab22c","timestamp":"2024-02-02T22:21:31Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:32 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3be3470e-9374-49f6-b64d-7aefeedab22c] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"3be3470e-9374-49f6-b64d-7aefeedab22c","timestamp":"2024-02-02T22:21:32Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === RUN   TestUpgrade/RetryFailureUpgrade[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === PAUSE TestUpgrade/RetryFailureUpgrade[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === RUN   TestUpgrade/RetryFailureUpgradeBackwardCompatible[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === PAUSE TestUpgrade/RetryFailureUpgradeBackwardCompatible[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === CONT  TestUpgrade/RetryFailureUpgradeBackwardCompatible[0m
2024-02-02 22:22:18 - INFO-root::util|431:: === CONT  TestUpgrade/RetryFailureUpgrade[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=0d248f0c-a19a-481a-b548-44f37bc0a188] could not obtain default workload version from configuration variables in DB","requestId":"0d248f0c-a19a-481a-b548-44f37bc0a188","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=b780b6aa-3cf4-492c-9948-2f624d44d4e7] could not obtain default workload version from configuration variables in DB","requestId":"b780b6aa-3cf4-492c-9948-2f624d44d4e7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=051ea301-a573-40b1-a1b0-f15e2fc5bfa0] could not obtain default workload version from configuration variables in DB","requestId":"051ea301-a573-40b1-a1b0-f15e2fc5bfa0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=0dc025df-d524-4c72-9a54-1e2f751167c7] could not obtain default workload version from configuration variables in DB","requestId":"0dc025df-d524-4c72-9a54-1e2f751167c7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=2512ce76-64c6-4092-b840-d4a67c159220] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"2512ce76-64c6-4092-b840-d4a67c159220","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=2512ce76-64c6-4092-b840-d4a67c159220] Pre-flight validation Category: 'FAILED', Name: 'Modify cluster security request validation', Message: ''Restrict access to Kubernetes API server to authorized IP ranges' knob cannot be enabled for 'Private cluster''.","requestId":"2512ce76-64c6-4092-b840-d4a67c159220","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:244","message":"[requestId=2512ce76-64c6-4092-b840-d4a67c159220] Failed to start ModifyClusterSecurity: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Modify cluster security request validation","detailedMessage":"'Restrict access to Kubernetes API server to authorized IP ranges' knob cannot be enabled for 'Private cluster'"}],"requestID":"2512ce76-64c6-4092-b840-d4a67c159220"}
","requestId":"2512ce76-64c6-4092-b840-d4a67c159220","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4973","message":"[requestId=0d248f0c-a19a-481a-b548-44f37bc0a188] Workspace version already higher than requested version","requestId":"0d248f0c-a19a-481a-b548-44f37bc0a188","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4955","message":"[requestId=b780b6aa-3cf4-492c-9948-2f624d44d4e7] cannot determine target Kubernetes version","requestId":"b780b6aa-3cf4-492c-9948-2f624d44d4e7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to start Upgrade: cannot determine target Kubernetes version[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to start Upgrade: Workspace version already higher than requested version[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=525aaa4e-f628-4e94-b3ff-481121d99677] could not obtain default workload version from configuration variables in DB","requestId":"525aaa4e-f628-4e94-b3ff-481121d99677","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4973","message":"[requestId=525aaa4e-f628-4e94-b3ff-481121d99677] Workspace version already higher than requested version","requestId":"525aaa4e-f628-4e94-b3ff-481121d99677","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=051ea301-a573-40b1-a1b0-f15e2fc5bfa0] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"051ea301-a573-40b1-a1b0-f15e2fc5bfa0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to start Upgrade: Workspace version already higher than requested version[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=74651c4a-c6ac-47dd-92b0-97f96c1a5a68] could not obtain default workload version from configuration variables in DB","requestId":"74651c4a-c6ac-47dd-92b0-97f96c1a5a68","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5037","message":"[requestId=051ea301-a573-40b1-a1b0-f15e2fc5bfa0] Workspace upgrade error handler is called","requestId":"051ea301-a573-40b1-a1b0-f15e2fc5bfa0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5038","message":"[requestId=051ea301-a573-40b1-a1b0-f15e2fc5bfa0] Failed to upgrade: Broken helm client does nothing
","requestId":"051ea301-a573-40b1-a1b0-f15e2fc5bfa0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=051ea301-a573-40b1-a1b0-f15e2fc5bfa0] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"051ea301-a573-40b1-a1b0-f15e2fc5bfa0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=051ea301-a573-40b1-a1b0-f15e2fc5bfa0] {"id":"a3baa0f9-0358-4ac5-86a7-5e28dd2bda84","timestamp":1706912495139,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:6d58d039-7f0e-4318-a419-e45e14a391fa","oldStatus":20,"newStatus":22,"failureReason":"Broken helm client does nothing","fromVersion":"2.0.32.0","toVersion":"2.0.34.0-skip-registry-migration-for-test","requestId":"051ea301-a573-40b1-a1b0-f15e2fc5bfa0","userTags":"{}","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"051ea301-a573-40b1-a1b0-f15e2fc5bfa0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=051ea301-a573-40b1-a1b0-f15e2fc5bfa0] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"UPGRADE_FAILED","OldStatus":"UPGRADE_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:6d58d039-7f0e-4318-a419-e45e14a391fa"}'","requestId":"051ea301-a573-40b1-a1b0-f15e2fc5bfa0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to execute upgrade: Broken helm client does nothing[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=0dc025df-d524-4c72-9a54-1e2f751167c7] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"0dc025df-d524-4c72-9a54-1e2f751167c7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=3085c184-80ca-435d-95a5-565a819eacc0] could not obtain default workload version from configuration variables in DB","requestId":"3085c184-80ca-435d-95a5-565a819eacc0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5037","message":"[requestId=0dc025df-d524-4c72-9a54-1e2f751167c7] Workspace upgrade error handler is called","requestId":"0dc025df-d524-4c72-9a54-1e2f751167c7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5038","message":"[requestId=0dc025df-d524-4c72-9a54-1e2f751167c7] Failed to upgrade: Broken helm client does nothing
","requestId":"0dc025df-d524-4c72-9a54-1e2f751167c7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=74651c4a-c6ac-47dd-92b0-97f96c1a5a68] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"74651c4a-c6ac-47dd-92b0-97f96c1a5a68","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to start Upgrade: Pre-flight validations failed. {"failedValidations":[{"category":"WORKSPACE_VERSION_SAME","name":"Workspace version validation","detailedMessage":"Workspace version already same as requested version"}],"requestID":"74651c4a-c6ac-47dd-92b0-97f96c1a5a68"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4910","message":"[requestId=3085c184-80ca-435d-95a5-565a819eacc0] workspace status: upgrade:failed <nil>","requestId":"3085c184-80ca-435d-95a5-565a819eacc0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=0dc025df-d524-4c72-9a54-1e2f751167c7] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"0dc025df-d524-4c72-9a54-1e2f751167c7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=0dc025df-d524-4c72-9a54-1e2f751167c7] {"id":"d2a4cf76-0d85-4d19-a3fc-e6c5831493ad","timestamp":1706912495162,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:284ea14f-92d1-4494-b259-af6c8c6b2dbc","oldStatus":20,"newStatus":22,"failureReason":"Broken helm client does nothing","fromVersion":"2.0.32.0","toVersion":"2.0.34.0-skip-registry-migration-for-test","requestId":"0dc025df-d524-4c72-9a54-1e2f751167c7","userTags":"{}","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"0dc025df-d524-4c72-9a54-1e2f751167c7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=0dc025df-d524-4c72-9a54-1e2f751167c7] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"UPGRADE_FAILED","OldStatus":"UPGRADE_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:284ea14f-92d1-4494-b259-af6c8c6b2dbc"}'","requestId":"0dc025df-d524-4c72-9a54-1e2f751167c7","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to execute upgrade: Broken helm client does nothing[0m
2024-02-02 22:22:18 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=76b83747-f86f-4eed-926b-99f12e458b5d] could not obtain default workload version from configuration variables in DB","requestId":"76b83747-f86f-4eed-926b-99f12e458b5d","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4910","message":"[requestId=76b83747-f86f-4eed-926b-99f12e458b5d] workspace status: installation:failed <nil>","requestId":"76b83747-f86f-4eed-926b-99f12e458b5d","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3085c184-80ca-435d-95a5-565a819eacc0] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"3085c184-80ca-435d-95a5-565a819eacc0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:18 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=3085c184-80ca-435d-95a5-565a819eacc0] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"UPGRADE_STARTED","OldStatus":"UPGRADE_FAILED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:6d58d039-7f0e-4318-a419-e45e14a391fa"}'","requestId":"3085c184-80ca-435d-95a5-565a819eacc0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=76b83747-f86f-4eed-926b-99f12e458b5d] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"76b83747-f86f-4eed-926b-99f12e458b5d","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=76b83747-f86f-4eed-926b-99f12e458b5d] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"UPGRADE_STARTED","OldStatus":"INSTALLATION_FAILED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:284ea14f-92d1-4494-b259-af6c8c6b2dbc"}'","requestId":"76b83747-f86f-4eed-926b-99f12e458b5d","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=3085c184-80ca-435d-95a5-565a819eacc0] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"3085c184-80ca-435d-95a5-565a819eacc0","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: 2024/02/02 22:21:35 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=76b83747-f86f-4eed-926b-99f12e458b5d] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"76b83747-f86f-4eed-926b-99f12e458b5d","timestamp":"2024-02-02T22:21:35Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431::     --- PASS: TestUpgrade/RetryFailureUpgrade (2.66s)[0m
2024-02-02 22:22:19 - INFO-root::util|431::     --- PASS: TestUpgrade/RetryFailureUpgradeBackwardCompatible (2.67s)[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === RUN   TestSuspend/Preflight_failure[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === PAUSE TestSuspend/Preflight_failure[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === CONT  TestSuspend/Preflight_failure[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === RUN   TestSuspend/PreconditionCheck/installation:failed[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=81c7a98d-65e3-442b-bd62-7a18dc171203] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"81c7a98d-65e3-442b-bd62-7a18dc171203","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"validator/validator.go:1673","message":"[requestId=9a35291f-2339-440a-98a4-ddbaa64820f3] ","requestId":"9a35291f-2339-440a-98a4-ddbaa64820f3","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cdpcontext/cdpcontext.go:239","message":"[requestId=9a35291f-2339-440a-98a4-ddbaa64820f3] Failed to write event msg AuditEvent needs to be created before adding log messages to it","requestId":"9a35291f-2339-440a-98a4-ddbaa64820f3","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4622","message":"[requestId=9a35291f-2339-440a-98a4-ddbaa64820f3] Failed to write event for pre-flight validation failure.","requestId":"9a35291f-2339-440a-98a4-ddbaa64820f3","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=9a35291f-2339-440a-98a4-ddbaa64820f3] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"9a35291f-2339-440a-98a4-ddbaa64820f3","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=9a35291f-2339-440a-98a4-ddbaa64820f3] Pre-flight validation Category: 'FAILED', Name: 'Suspend Workspace Validation', Message: 'There are some infra jobs running. Please retry the suspend operation after sometime'.","requestId":"9a35291f-2339-440a-98a4-ddbaa64820f3","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=9a35291f-2339-440a-98a4-ddbaa64820f3] Failed to start Suspend: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Suspend Workspace Validation","detailedMessage":"There are some infra jobs running. Please retry the suspend operation after sometime"}],"requestID":"9a35291f-2339-440a-98a4-ddbaa64820f3"}
","requestId":"9a35291f-2339-440a-98a4-ddbaa64820f3","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:560","message":"[requestId=6a029cff-b82e-4767-806a-f1e96a81e7db] The preflight check for FreeIPA failed: &{ <nil>   <nil> <nil> <nil> <nil> <nil> [] <nil> <nil> <nil> UNAVAILABLE   <nil>  <nil> }","requestId":"6a029cff-b82e-4767-806a-f1e96a81e7db","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7559","message":"[requestId=6a029cff-b82e-4767-806a-f1e96a81e7db] Preflight check for suspend workspace failed. This environment free IPA is not ready","requestId":"6a029cff-b82e-4767-806a-f1e96a81e7db","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=6a029cff-b82e-4767-806a-f1e96a81e7db] Failed to start Suspend: This environment free IPA is not ready
","requestId":"6a029cff-b82e-4767-806a-f1e96a81e7db","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:526","message":"[requestId=45a82e68-5542-46aa-9fc4-e6426252a531] The preflight check for Cloudbreak failed:  UNAVAILABLE","requestId":"45a82e68-5542-46aa-9fc4-e6426252a531","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7559","message":"[requestId=45a82e68-5542-46aa-9fc4-e6426252a531] Preflight check for suspend workspace failed. This environment is not ready","requestId":"45a82e68-5542-46aa-9fc4-e6426252a531","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=45a82e68-5542-46aa-9fc4-e6426252a531] Failed to start Suspend: This environment is not ready
","requestId":"45a82e68-5542-46aa-9fc4-e6426252a531","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:542","message":"[requestId=d45c48b8-e410-4943-b15b-6937d92401cf] The preflight check for Datalake failed: &{    0   false false   <nil>  false    UNAVAILABLE  map[]}","requestId":"d45c48b8-e410-4943-b15b-6937d92401cf","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7559","message":"[requestId=d45c48b8-e410-4943-b15b-6937d92401cf] Preflight check for suspend workspace failed. This environment datalake is not ready","requestId":"d45c48b8-e410-4943-b15b-6937d92401cf","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d45c48b8-e410-4943-b15b-6937d92401cf] Failed to start Suspend: This environment datalake is not ready
","requestId":"d45c48b8-e410-4943-b15b-6937d92401cf","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=1907fb4f-9189-4858-8721-0542090cef3d] Failed to start Suspend: required entitlements not present for this operation
","requestId":"1907fb4f-9189-4858-8721-0542090cef3d","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d582e00e-fd8e-4b70-be0a-ddbd1d60028a] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"d582e00e-fd8e-4b70-be0a-ddbd1d60028a","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=3dd85b7b-3d43-4d95-98e0-e7aecb04da99] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"3dd85b7b-3d43-4d95-98e0-e7aecb04da99","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=4f72e873-2a26-4d63-a7fb-ccf64a83ea7a] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"4f72e873-2a26-4d63-a7fb-ccf64a83ea7a","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=5e2aeb40-6c37-4b25-9c69-0582bcbf6324] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"5e2aeb40-6c37-4b25-9c69-0582bcbf6324","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=e8eb2994-8256-4a24-b4f1-500aae9c7e14] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"e8eb2994-8256-4a24-b4f1-500aae9c7e14","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=94afcefe-3436-4cbd-8629-7229dbc672fd] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"94afcefe-3436-4cbd-8629-7229dbc672fd","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=85de8709-28d0-4eb4-ad85-b83289e6fa56] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"85de8709-28d0-4eb4-ad85-b83289e6fa56","timestamp":"2024-02-02T22:21:36Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=6fbfa6c0-eb17-448d-9c81-c388d237790e] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"6fbfa6c0-eb17-448d-9c81-c388d237790e","timestamp":"2024-02-02T22:21:37Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=6c3593d6-82bf-4089-9ede-8b5813811362] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"6c3593d6-82bf-4089-9ede-8b5813811362","timestamp":"2024-02-02T22:21:37Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=a3fabfcd-bcc7-482e-a11c-1e04ef2cf2b3] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"a3fabfcd-bcc7-482e-a11c-1e04ef2cf2b3","timestamp":"2024-02-02T22:21:37Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=c057e68b-bfaf-45a2-849e-17aa28776f62] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"c057e68b-bfaf-45a2-849e-17aa28776f62","timestamp":"2024-02-02T22:21:37Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=8bd8534e-b28c-4689-be5a-2377758935ad] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"8bd8534e-b28c-4689-be5a-2377758935ad","timestamp":"2024-02-02T22:21:37Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=04c57d8e-b2b0-4fc8-9a0f-a4ebaaf3a983] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"04c57d8e-b2b0-4fc8-9a0f-a4ebaaf3a983","timestamp":"2024-02-02T22:21:37Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=752a85af-d12c-4400-9a07-aef7ca1c1af4] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"752a85af-d12c-4400-9a07-aef7ca1c1af4","timestamp":"2024-02-02T22:21:37Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=00a2630e-2eb9-412b-9ba5-bdfbf15e05a5] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"00a2630e-2eb9-412b-9ba5-bdfbf15e05a5","timestamp":"2024-02-02T22:21:37Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=e920c1de-ee53-4abc-ba4d-82637300ff64] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"e920c1de-ee53-4abc-ba4d-82637300ff64","timestamp":"2024-02-02T22:21:38Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d7c4593b-343a-488d-9e48-015811d3d184] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"d7c4593b-343a-488d-9e48-015811d3d184","timestamp":"2024-02-02T22:21:38Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=89fe23b7-b276-4fb1-98e1-cef1fdc9815b] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"89fe23b7-b276-4fb1-98e1-cef1fdc9815b","timestamp":"2024-02-02T22:21:38Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=4bda7b23-e9c6-4eff-b8bb-1f5a7dd60fc4] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"4bda7b23-e9c6-4eff-b8bb-1f5a7dd60fc4","timestamp":"2024-02-02T22:21:38Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=18d66acc-c9ae-40af-9485-383075ad4b01] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"18d66acc-c9ae-40af-9485-383075ad4b01","timestamp":"2024-02-02T22:21:38Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=6fb8cccf-b551-4139-9273-3549329cf221] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"6fb8cccf-b551-4139-9273-3549329cf221","timestamp":"2024-02-02T22:21:38Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=79276a11-21d4-4131-a623-4f1c2c3cb9d0] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"79276a11-21d4-4131-a623-4f1c2c3cb9d0","timestamp":"2024-02-02T22:21:39Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=1f25ebf5-40fc-4d31-8e4c-ba5fc15252e3] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"1f25ebf5-40fc-4d31-8e4c-ba5fc15252e3","timestamp":"2024-02-02T22:21:39Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d2bb997a-7cc4-4937-95a5-acc4153aeacd] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"d2bb997a-7cc4-4937-95a5-acc4153aeacd","timestamp":"2024-02-02T22:21:39Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=5db85db8-4a42-42d8-9999-048bec1d1287] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"5db85db8-4a42-42d8-9999-048bec1d1287","timestamp":"2024-02-02T22:21:39Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=ee208878-7f11-4084-9d4c-7bf0a48a10ba] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"ee208878-7f11-4084-9d4c-7bf0a48a10ba","timestamp":"2024-02-02T22:21:39Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=b7ce5187-973b-4f08-8624-3d3c2cb55394] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"b7ce5187-973b-4f08-8624-3d3c2cb55394","timestamp":"2024-02-02T22:21:39Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=938ecf4e-5422-4f88-8d34-7b52385dcaf2] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"938ecf4e-5422-4f88-8d34-7b52385dcaf2","timestamp":"2024-02-02T22:21:39Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=c159009e-59e2-4c96-bbeb-d7c7ffb771da] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"c159009e-59e2-4c96-bbeb-d7c7ffb771da","timestamp":"2024-02-02T22:21:40Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d321cdfe-2597-48d3-be72-5d187b893b44] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"d321cdfe-2597-48d3-be72-5d187b893b44","timestamp":"2024-02-02T22:21:40Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=66fe8a62-a9a0-4545-9c8b-f2ae66a55639] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"66fe8a62-a9a0-4545-9c8b-f2ae66a55639","timestamp":"2024-02-02T22:21:40Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=0641a052-a2d0-4b8e-8c76-f6c788f2b303] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"0641a052-a2d0-4b8e-8c76-f6c788f2b303","timestamp":"2024-02-02T22:21:40Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d32359ba-7a01-4bab-b422-8b577acdb8fe] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"d32359ba-7a01-4bab-b422-8b577acdb8fe","timestamp":"2024-02-02T22:21:40Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=c53e1d32-01d5-4dc9-adaa-89bba68a7a0b] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"c53e1d32-01d5-4dc9-adaa-89bba68a7a0b","timestamp":"2024-02-02T22:21:40Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=7d8e76e1-2427-40a6-9faa-e3e7fbf33207] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"7d8e76e1-2427-40a6-9faa-e3e7fbf33207","timestamp":"2024-02-02T22:21:41Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=cba6415c-d124-4f47-a116-53cbd6c31107] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"cba6415c-d124-4f47-a116-53cbd6c31107","timestamp":"2024-02-02T22:21:41Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=cbc03908-d6bf-4741-9b39-e5509fa32846] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"cbc03908-d6bf-4741-9b39-e5509fa32846","timestamp":"2024-02-02T22:21:41Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=b41dd94d-1988-4d67-b529-d98edef15049] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"b41dd94d-1988-4d67-b529-d98edef15049","timestamp":"2024-02-02T22:21:41Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=b7a8df0d-09d9-4ab3-a5ea-47aa6f2172b4] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"b7a8df0d-09d9-4ab3-a5ea-47aa6f2172b4","timestamp":"2024-02-02T22:21:41Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=309f77cd-2dc0-4bea-a82c-2d11d0981fc5] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"309f77cd-2dc0-4bea-a82c-2d11d0981fc5","timestamp":"2024-02-02T22:21:41Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=2022f480-24ce-4b1f-a048-f908accf2503] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"2022f480-24ce-4b1f-a048-f908accf2503","timestamp":"2024-02-02T22:21:41Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=3a3c206d-d0d9-4b2d-ba35-e8fa8510a517] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"3a3c206d-d0d9-4b2d-ba35-e8fa8510a517","timestamp":"2024-02-02T22:21:42Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=47b126c6-81f5-42ac-8572-c014aa9aa16b] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"47b126c6-81f5-42ac-8572-c014aa9aa16b","timestamp":"2024-02-02T22:21:42Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=22304065-ac8d-4a3b-a1d7-2c45cc57e372] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"22304065-ac8d-4a3b-a1d7-2c45cc57e372","timestamp":"2024-02-02T22:21:42Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=aab0ec88-5236-4bf1-9617-ae61f7dcc81a] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"aab0ec88-5236-4bf1-9617-ae61f7dcc81a","timestamp":"2024-02-02T22:21:42Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=e365f90d-c862-474e-927f-e7f022b6095e] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"e365f90d-c862-474e-927f-e7f022b6095e","timestamp":"2024-02-02T22:21:42Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=2c370e79-79bf-4afb-bd63-bed7007f749e] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"2c370e79-79bf-4afb-bd63-bed7007f749e","timestamp":"2024-02-02T22:21:42Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=10484d6b-81b0-4eee-836e-1d0d215ccafb] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"10484d6b-81b0-4eee-836e-1d0d215ccafb","timestamp":"2024-02-02T22:21:42Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=e1670928-4c84-4763-ac6c-5de4db4cf88d] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"e1670928-4c84-4763-ac6c-5de4db4cf88d","timestamp":"2024-02-02T22:21:42Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=b9f083d1-366c-4720-a7cb-ecf40d8c89cc] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"b9f083d1-366c-4720-a7cb-ecf40d8c89cc","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=74888e37-5908-4306-b031-8563ac9d3f19] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"74888e37-5908-4306-b031-8563ac9d3f19","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=5e90a9ed-93dd-454f-ba50-b5ddf295fde3] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"5e90a9ed-93dd-454f-ba50-b5ddf295fde3","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=0a26a68a-4795-4105-b2dc-47c353853793] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"0a26a68a-4795-4105-b2dc-47c353853793","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=506078f5-0bad-48dc-9590-06d2ed62b423] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"506078f5-0bad-48dc-9590-06d2ed62b423","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=c20ebf0e-0f2b-4ffe-bbfb-5b349aaa8fa2] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"c20ebf0e-0f2b-4ffe-bbfb-5b349aaa8fa2","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=7b65e476-c754-4ba8-8837-aabefcb89fb2] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"7b65e476-c754-4ba8-8837-aabefcb89fb2","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=545dafbe-df2f-4d74-93d9-6a6bac436f08] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"545dafbe-df2f-4d74-93d9-6a6bac436f08","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=69b00b54-6503-4581-8e8c-fb2e94319b6d] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"69b00b54-6503-4581-8e8c-fb2e94319b6d","timestamp":"2024-02-02T22:21:43Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=f997cbb3-59e5-4a55-a72b-9eb80605e664] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"f997cbb3-59e5-4a55-a72b-9eb80605e664","timestamp":"2024-02-02T22:21:44Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=05c27f1c-e23d-48cc-b862-2d385018029b] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"05c27f1c-e23d-48cc-b862-2d385018029b","timestamp":"2024-02-02T22:21:44Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=7d56e34c-b658-461d-b6c8-f242fca0811d] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"7d56e34c-b658-461d-b6c8-f242fca0811d","timestamp":"2024-02-02T22:21:44Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=4e1bdd69-330b-4375-ab49-d3b3f9a03dfd] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"4e1bdd69-330b-4375-ab49-d3b3f9a03dfd","timestamp":"2024-02-02T22:21:44Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=63dbabee-7cff-41ca-904f-8b0a3de293d6] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"63dbabee-7cff-41ca-904f-8b0a3de293d6","timestamp":"2024-02-02T22:21:44Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d823bd13-5c43-461d-8461-721ef08d25e1] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"d823bd13-5c43-461d-8461-721ef08d25e1","timestamp":"2024-02-02T22:21:45Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=a7d10536-ada8-469b-aae3-5eb1f5b291f3] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"a7d10536-ada8-469b-aae3-5eb1f5b291f3","timestamp":"2024-02-02T22:21:45Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=df081282-4482-4d3a-9edb-647f29bf1b54] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"df081282-4482-4d3a-9edb-647f29bf1b54","timestamp":"2024-02-02T22:21:45Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=c7723024-1504-49e8-b8b3-557409c4aa4a] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"c7723024-1504-49e8-b8b3-557409c4aa4a","timestamp":"2024-02-02T22:21:45Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=14507d2a-82c3-4d3b-b32b-0e86aea099ee] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"14507d2a-82c3-4d3b-b32b-0e86aea099ee","timestamp":"2024-02-02T22:21:45Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=fcd004b3-1594-4c49-8e7e-b25ba95d92c0] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"fcd004b3-1594-4c49-8e7e-b25ba95d92c0","timestamp":"2024-02-02T22:21:45Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=bf2dba93-3655-48af-aefb-068f75391bdb] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"bf2dba93-3655-48af-aefb-068f75391bdb","timestamp":"2024-02-02T22:21:45Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=a009e870-99e7-4a26-b115-0a6ef9364089] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"a009e870-99e7-4a26-b115-0a6ef9364089","timestamp":"2024-02-02T22:21:45Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=cf5fc657-0f09-474b-852a-c9aaf3ce7408] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"cf5fc657-0f09-474b-852a-c9aaf3ce7408","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=441da2c3-2eaa-4966-88df-5a4af42b896a] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"441da2c3-2eaa-4966-88df-5a4af42b896a","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=9e9543c0-671f-4719-8417-f36590a28b78] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"9e9543c0-671f-4719-8417-f36590a28b78","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=00eb40f3-d875-4267-850d-06885fa907d7] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"00eb40f3-d875-4267-850d-06885fa907d7","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=bbc26cab-98b9-4bce-9362-7bd7dec2c9b3] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"bbc26cab-98b9-4bce-9362-7bd7dec2c9b3","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=4ac88eed-2420-46f7-b28e-2c91b406d314] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"4ac88eed-2420-46f7-b28e-2c91b406d314","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=08546ce5-2540-4a7e-b122-5b47917d0b6b] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"08546ce5-2540-4a7e-b122-5b47917d0b6b","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=2ebc030d-9cee-4292-816f-03bd83184078] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"2ebc030d-9cee-4292-816f-03bd83184078","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=3ca5529b-5be4-4d30-aca0-ab3ad572061c] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"3ca5529b-5be4-4d30-aca0-ab3ad572061c","timestamp":"2024-02-02T22:21:46Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=e2775624-ba19-408c-bf64-0aae1c3713e6] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"e2775624-ba19-408c-bf64-0aae1c3713e6","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=e6794fb7-b0d7-4d10-80ab-e2a7f40f6ec8] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"e6794fb7-b0d7-4d10-80ab-e2a7f40f6ec8","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d63177b2-7f12-4ac0-907d-aee7d64e8483] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"d63177b2-7f12-4ac0-907d-aee7d64e8483","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=5bc29b7a-0322-466c-9668-ce83061c948f] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"5bc29b7a-0322-466c-9668-ce83061c948f","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=4147defe-d746-430b-88c8-e6df3f5b092b] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"4147defe-d746-430b-88c8-e6df3f5b092b","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=933ee95c-9553-4749-bcdb-8efff2f8ccfb] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"933ee95c-9553-4749-bcdb-8efff2f8ccfb","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=edd15859-bb80-4b02-966c-4425ec3dc639] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"edd15859-bb80-4b02-966c-4425ec3dc639","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=603ba84c-8412-4b96-a418-c7616d6468f4] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"603ba84c-8412-4b96-a418-c7616d6468f4","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=b6b0297c-7803-4b01-8991-1f143f9c2d85] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"b6b0297c-7803-4b01-8991-1f143f9c2d85","timestamp":"2024-02-02T22:21:47Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=2e1f907c-33ca-43ca-8c04-5d8494cb8e94] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"2e1f907c-33ca-43ca-8c04-5d8494cb8e94","timestamp":"2024-02-02T22:21:48Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=a1c8459d-30a7-4b61-8a0c-8ddc995f880d] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"a1c8459d-30a7-4b61-8a0c-8ddc995f880d","timestamp":"2024-02-02T22:21:48Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=814ec8a2-4403-49b1-9cc2-513fff581004] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"814ec8a2-4403-49b1-9cc2-513fff581004","timestamp":"2024-02-02T22:21:48Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=478c1dcf-55e7-465e-8aca-cadd2c28df0f] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"478c1dcf-55e7-465e-8aca-cadd2c28df0f","timestamp":"2024-02-02T22:21:48Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=941d4ef4-a086-4358-a3e8-b2d15c87b207] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"941d4ef4-a086-4358-a3e8-b2d15c87b207","timestamp":"2024-02-02T22:21:48Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=4db1b838-91c8-424a-8e1e-4cede4dd5396] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"4db1b838-91c8-424a-8e1e-4cede4dd5396","timestamp":"2024-02-02T22:21:48Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=31080ef0-dc74-4db7-9df4-d2779a337ebd] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"31080ef0-dc74-4db7-9df4-d2779a337ebd","timestamp":"2024-02-02T22:21:48Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=0c7c7fc2-0bef-43e2-ab3d-98011ce58a1b] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"0c7c7fc2-0bef-43e2-ab3d-98011ce58a1b","timestamp":"2024-02-02T22:21:48Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=d7ad50c9-58a8-4695-be9c-bb34b02979f7] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"d7ad50c9-58a8-4695-be9c-bb34b02979f7","timestamp":"2024-02-02T22:21:49Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=f0465f3d-d090-42ff-bf3a-ce8a73ef91e5] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"f0465f3d-d090-42ff-bf3a-ce8a73ef91e5","timestamp":"2024-02-02T22:21:49Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=7fcc6960-cbf7-4708-9010-f187b45c3f3f] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"7fcc6960-cbf7-4708-9010-f187b45c3f3f","timestamp":"2024-02-02T22:21:49Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=06a747e6-321f-44a4-8b09-0879b8aa9e16] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"06a747e6-321f-44a4-8b09-0879b8aa9e16","timestamp":"2024-02-02T22:21:49Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=cf574fa8-6c35-49f6-bdfa-e932ac2f6669] Failed to start Suspend: Cannot suspend a workspace that is not ready
","requestId":"cf574fa8-6c35-49f6-bdfa-e932ac2f6669","timestamp":"2024-02-02T22:21:49Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431::         --- PASS: TestSuspend/PreconditionCheck/installation:failed (0.00s)[0m
2024-02-02 22:22:19 - INFO-root::util|431::     --- PASS: TestSuspend/Preflight_failure (0.73s)[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === RUN   TestResume/Preflight_failure[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === PAUSE TestResume/Preflight_failure[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === CONT  TestResume/Preflight_failure[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:147","message":"[requestId=0b259061-4a3a-49b1-b8ff-d0eefda42d8d] Failed to start Suspend: required entitlements not present for this operation
","requestId":"0b259061-4a3a-49b1-b8ff-d0eefda42d8d","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:560","message":"[requestId=a067355d-49c3-4fe3-8007-c603b9bffcde] The preflight check for FreeIPA failed: &{ <nil>   <nil> <nil> <nil> <nil> <nil> [] <nil> <nil> <nil> UNAVAILABLE   <nil>  <nil> }","requestId":"a067355d-49c3-4fe3-8007-c603b9bffcde","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7730","message":"[requestId=a067355d-49c3-4fe3-8007-c603b9bffcde] Preflight check for suspend workspace failed. This environment free IPA is not ready","requestId":"a067355d-49c3-4fe3-8007-c603b9bffcde","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:160","message":"[requestId=a067355d-49c3-4fe3-8007-c603b9bffcde] Failed to start Resume: This environment free IPA is not ready
","requestId":"a067355d-49c3-4fe3-8007-c603b9bffcde","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:526","message":"[requestId=71493509-7d98-4c46-8bd4-4810a22759ec] The preflight check for Cloudbreak failed:  UNAVAILABLE","requestId":"71493509-7d98-4c46-8bd4-4810a22759ec","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7730","message":"[requestId=71493509-7d98-4c46-8bd4-4810a22759ec] Preflight check for suspend workspace failed. This environment is not ready","requestId":"71493509-7d98-4c46-8bd4-4810a22759ec","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:160","message":"[requestId=71493509-7d98-4c46-8bd4-4810a22759ec] Failed to start Resume: This environment is not ready
","requestId":"71493509-7d98-4c46-8bd4-4810a22759ec","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:542","message":"[requestId=0a7467d7-bee7-45c2-9fe9-8d2c1143f37b] The preflight check for Datalake failed: &{    0   false false   <nil>  false    UNAVAILABLE  map[]}","requestId":"0a7467d7-bee7-45c2-9fe9-8d2c1143f37b","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7730","message":"[requestId=0a7467d7-bee7-45c2-9fe9-8d2c1143f37b] Preflight check for suspend workspace failed. This environment datalake is not ready","requestId":"0a7467d7-bee7-45c2-9fe9-8d2c1143f37b","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:160","message":"[requestId=0a7467d7-bee7-45c2-9fe9-8d2c1143f37b] Failed to start Resume: This environment datalake is not ready
","requestId":"0a7467d7-bee7-45c2-9fe9-8d2c1143f37b","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431::     --- PASS: TestResume/Preflight_failure (0.85s)[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cdpcontext/cdpcontext.go:239","message":"[requestId=53e73af0-6df7-4812-b6fb-eb9d4b167ebd] Failed to write event msg AuditEvent needs to be created before adding log messages to it","requestId":"53e73af0-6df7-4812-b6fb-eb9d4b167ebd","timestamp":"2024-02-02T22:21:50Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: 2024/02/02 22:21:50 Failed to start CreateClusterInstanceGroup: required entitlements not present for this operation[0m
2024-02-02 22:22:19 - INFO-root::util|431:: 2024/02/02 22:21:51 Failed to start CreateClusterInstanceGroup: workspace not in a modifiable state[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:232","message":"[requestId=a20dfded-ee7f-4734-bfb9-81fae26cabb1] Failed to DeleteInstanceGroup: required entitlements not present for this operation
","requestId":"a20dfded-ee7f-4734-bfb9-81fae26cabb1","timestamp":"2024-02-02T22:21:51Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cdpcontext/cdpcontext.go:239","message":"[requestId=8bd756e8-57c7-4ca6-8815-12399809a779] Failed to write event msg AuditEvent needs to be created before adding log messages to it","requestId":"8bd756e8-57c7-4ca6-8815-12399809a779","timestamp":"2024-02-02T22:21:51Z"}[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === RUN   TestModifyCluster/Preflight_failure_Modify_Instance_Group[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === PAUSE TestModifyCluster/Preflight_failure_Modify_Instance_Group[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === RUN   TestModifyCluster/Preflight_failure_Modify_Security[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === PAUSE TestModifyCluster/Preflight_failure_Modify_Security[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === RUN   TestModifyCluster/Preflight_failure_Modify_LB_Source[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === PAUSE TestModifyCluster/Preflight_failure_Modify_LB_Source[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === CONT  TestModifyCluster/Preflight_failure_Modify_Security[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === CONT  TestModifyCluster/Preflight_failure_Modify_LB_Source[0m
2024-02-02 22:22:19 - INFO-root::util|431:: === CONT  TestModifyCluster/Preflight_failure_Modify_Instance_Group[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:560","message":"[requestId=829e1a79-b8ea-4b0d-b8b4-fe7f55b620dc] The preflight check for FreeIPA failed: &{ <nil>   <nil> <nil> <nil> <nil> <nil> [] <nil> <nil> <nil> UNAVAILABLE   <nil>  <nil> }","requestId":"829e1a79-b8ea-4b0d-b8b4-fe7f55b620dc","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6905","message":"[requestId=829e1a79-b8ea-4b0d-b8b4-fe7f55b620dc] Preflight check for modify workspace load balancer failed. This environment free IPA is not ready","requestId":"829e1a79-b8ea-4b0d-b8b4-fe7f55b620dc","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:262","message":"[requestId=829e1a79-b8ea-4b0d-b8b4-fe7f55b620dc] Failed to execute ModifyWorkspaceLoadBalancer: This environment free IPA is not ready
","requestId":"829e1a79-b8ea-4b0d-b8b4-fe7f55b620dc","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:526","message":"[requestId=f466760f-6a05-42c7-a0e2-99a107f1a8b3] The preflight check for Cloudbreak failed:  UNAVAILABLE","requestId":"f466760f-6a05-42c7-a0e2-99a107f1a8b3","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6905","message":"[requestId=f466760f-6a05-42c7-a0e2-99a107f1a8b3] Preflight check for modify workspace load balancer failed. This environment is not ready","requestId":"f466760f-6a05-42c7-a0e2-99a107f1a8b3","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:262","message":"[requestId=f466760f-6a05-42c7-a0e2-99a107f1a8b3] Failed to execute ModifyWorkspaceLoadBalancer: This environment is not ready
","requestId":"f466760f-6a05-42c7-a0e2-99a107f1a8b3","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:542","message":"[requestId=622df1e3-42d6-4e28-a4e1-5a68d26de6fd] The preflight check for Datalake failed: &{    0   false false   <nil>  false    UNAVAILABLE  map[]}","requestId":"622df1e3-42d6-4e28-a4e1-5a68d26de6fd","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6905","message":"[requestId=622df1e3-42d6-4e28-a4e1-5a68d26de6fd] Preflight check for modify workspace load balancer failed. This environment datalake is not ready","requestId":"622df1e3-42d6-4e28-a4e1-5a68d26de6fd","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:262","message":"[requestId=622df1e3-42d6-4e28-a4e1-5a68d26de6fd] Failed to execute ModifyWorkspaceLoadBalancer: This environment datalake is not ready
","requestId":"622df1e3-42d6-4e28-a4e1-5a68d26de6fd","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:560","message":"[requestId=0cb9b37c-bbac-43ad-bad2-46d96100062a] The preflight check for FreeIPA failed: &{ <nil>   <nil> <nil> <nil> <nil> <nil> [] <nil> <nil> <nil> UNAVAILABLE   <nil>  <nil> }","requestId":"0cb9b37c-bbac-43ad-bad2-46d96100062a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6988","message":"[requestId=0cb9b37c-bbac-43ad-bad2-46d96100062a] Preflight check for modify cluster security group failed. This environment free IPA is not ready","requestId":"0cb9b37c-bbac-43ad-bad2-46d96100062a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:244","message":"[requestId=0cb9b37c-bbac-43ad-bad2-46d96100062a] Failed to start ModifyClusterSecurity: This environment free IPA is not ready
","requestId":"0cb9b37c-bbac-43ad-bad2-46d96100062a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:526","message":"[requestId=ad287986-6a2f-494c-9a5e-4b4a05771598] The preflight check for Cloudbreak failed:  UNAVAILABLE","requestId":"ad287986-6a2f-494c-9a5e-4b4a05771598","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6988","message":"[requestId=ad287986-6a2f-494c-9a5e-4b4a05771598] Preflight check for modify cluster security group failed. This environment is not ready","requestId":"ad287986-6a2f-494c-9a5e-4b4a05771598","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:244","message":"[requestId=ad287986-6a2f-494c-9a5e-4b4a05771598] Failed to start ModifyClusterSecurity: This environment is not ready
","requestId":"ad287986-6a2f-494c-9a5e-4b4a05771598","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:542","message":"[requestId=388558f6-b8ad-46b7-bf48-90a7fe4b18c2] The preflight check for Datalake failed: &{    0   false false   <nil>  false    UNAVAILABLE  map[]}","requestId":"388558f6-b8ad-46b7-bf48-90a7fe4b18c2","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6988","message":"[requestId=388558f6-b8ad-46b7-bf48-90a7fe4b18c2] Preflight check for modify cluster security group failed. This environment datalake is not ready","requestId":"388558f6-b8ad-46b7-bf48-90a7fe4b18c2","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:244","message":"[requestId=388558f6-b8ad-46b7-bf48-90a7fe4b18c2] Failed to start ModifyClusterSecurity: This environment datalake is not ready
","requestId":"388558f6-b8ad-46b7-bf48-90a7fe4b18c2","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cdpcontext/cdpcontext.go:239","message":"[requestId=3fb53390-03c9-4275-98e9-45369fe1d091] Failed to write event msg AuditEvent needs to be created before adding log messages to it","requestId":"3fb53390-03c9-4275-98e9-45369fe1d091","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:560","message":"[requestId=3fb53390-03c9-4275-98e9-45369fe1d091] The preflight check for FreeIPA failed: &{ <nil>   <nil> <nil> <nil> <nil> <nil> [] <nil> <nil> <nil> UNAVAILABLE   <nil>  <nil> }","requestId":"3fb53390-03c9-4275-98e9-45369fe1d091","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7419","message":"[requestId=3fb53390-03c9-4275-98e9-45369fe1d091] Preflight check for modify cluster instance group failed. This environment free IPA is not ready","requestId":"3fb53390-03c9-4275-98e9-45369fe1d091","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:209","message":"[requestId=3fb53390-03c9-4275-98e9-45369fe1d091] Failed to start ModifyClusterInstanceGroup: This environment free IPA is not ready
","requestId":"3fb53390-03c9-4275-98e9-45369fe1d091","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cdpcontext/cdpcontext.go:239","message":"[requestId=25024225-2efc-4d69-97b2-77c41e0fdc8c] Failed to write event msg AuditEvent needs to be created before adding log messages to it","requestId":"25024225-2efc-4d69-97b2-77c41e0fdc8c","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:526","message":"[requestId=25024225-2efc-4d69-97b2-77c41e0fdc8c] The preflight check for Cloudbreak failed:  UNAVAILABLE","requestId":"25024225-2efc-4d69-97b2-77c41e0fdc8c","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7419","message":"[requestId=25024225-2efc-4d69-97b2-77c41e0fdc8c] Preflight check for modify cluster instance group failed. This environment is not ready","requestId":"25024225-2efc-4d69-97b2-77c41e0fdc8c","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:209","message":"[requestId=25024225-2efc-4d69-97b2-77c41e0fdc8c] Failed to start ModifyClusterInstanceGroup: This environment is not ready
","requestId":"25024225-2efc-4d69-97b2-77c41e0fdc8c","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cdpcontext/cdpcontext.go:239","message":"[requestId=e1677b71-4443-4eaa-8de9-5132b10ac505] Failed to write event msg AuditEvent needs to be created before adding log messages to it","requestId":"e1677b71-4443-4eaa-8de9-5132b10ac505","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"core/core.go:542","message":"[requestId=e1677b71-4443-4eaa-8de9-5132b10ac505] The preflight check for Datalake failed: &{    0   false false   <nil>  false    UNAVAILABLE  map[]}","requestId":"e1677b71-4443-4eaa-8de9-5132b10ac505","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7419","message":"[requestId=e1677b71-4443-4eaa-8de9-5132b10ac505] Preflight check for modify cluster instance group failed. This environment datalake is not ready","requestId":"e1677b71-4443-4eaa-8de9-5132b10ac505","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:209","message":"[requestId=e1677b71-4443-4eaa-8de9-5132b10ac505] Failed to start ModifyClusterInstanceGroup: This environment datalake is not ready
","requestId":"e1677b71-4443-4eaa-8de9-5132b10ac505","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: 2024/02/02 22:21:52 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=d9bed5bb-a82c-4341-a7b5-0868f8aa1d4d] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"d9bed5bb-a82c-4341-a7b5-0868f8aa1d4d","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cdpcontext/cdpcontext.go:239","message":"[requestId=9b0a3a8b-469d-484a-95ff-b1729d98b9af] Failed to write event msg AuditEvent needs to be created before adding log messages to it","requestId":"9b0a3a8b-469d-484a-95ff-b1729d98b9af","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: 2024/02/02 22:21:52 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7045","message":"[requestId=6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a] Workspace update cluster security error handler is called","requestId":"6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:7046","message":"[requestId=6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a] Update cluster security failed: Broken provisioner does nothing
","requestId":"6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: 2024/02/02 22:21:52 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a] {"id":"fe372f01-77a3-4b35-9192-060756c02279","timestamp":1706912512351,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:75ac0ed4-ac35-46e0-8cd7-ec4cf5306c7b","oldStatus":33,"newStatus":35,"failureReason":"Broken provisioner does nothing","requestId":"6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a","userTags":"{}","cdpCMLWorkspaceRestoreMetadata":{},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"MODIFY_FAILED","OldStatus":"MODIFY_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:75ac0ed4-ac35-46e0-8cd7-ec4cf5306c7b"}'","requestId":"6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:249","message":"[requestId=6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a] Failed to execute ModifyClusterSecurity: Broken provisioner does nothing
","requestId":"6d230cb6-2d92-431f-92ff-9b7f8b7d4b4a","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: 2024/02/02 22:21:52 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=d9bed5bb-a82c-4341-a7b5-0868f8aa1d4d] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"d9bed5bb-a82c-4341-a7b5-0868f8aa1d4d","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431::     --- PASS: TestModifyCluster/Preflight_failure_Modify_LB_Source (0.72s)[0m
2024-02-02 22:22:20 - INFO-root::util|431::     --- PASS: TestModifyCluster/Preflight_failure_Modify_Security (0.74s)[0m
2024-02-02 22:22:20 - INFO-root::util|431::     --- PASS: TestModifyCluster/Preflight_failure_Modify_Instance_Group (0.77s)[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5962","message":"[requestId=607a8646-a1f8-492e-a0e6-56f9fd946dbd] unexpected error","requestId":"607a8646-a1f8-492e-a0e6-56f9fd946dbd","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"validator/validator.go:1778","message":"[requestId=ec6160be-3d32-4b6c-b8f8-7192945a8de6] User role validation result: 'FAILED', message: 'User role validations failed.'","requestId":"ec6160be-3d32-4b6c-b8f8-7192945a8de6","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=ec6160be-3d32-4b6c-b8f8-7192945a8de6] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"ec6160be-3d32-4b6c-b8f8-7192945a8de6","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=ec6160be-3d32-4b6c-b8f8-7192945a8de6] Pre-flight validation Category: 'CML_USER_ROLE', Name: 'Role validation 'random_role'.', Message: 'Role validation failed. User must have role 'random_role' for this operation.'.","requestId":"ec6160be-3d32-4b6c-b8f8-7192945a8de6","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6026","message":"[requestId=ec6160be-3d32-4b6c-b8f8-7192945a8de6] Pre-flight validations failed. {"failedValidations":[{"category":"CML_USER_ROLE","name":"Role validation 'random_role'.","detailedMessage":"Role validation failed. User must have role 'random_role' for this operation."}],"requestID":"ec6160be-3d32-4b6c-b8f8-7192945a8de6"}","requestId":"ec6160be-3d32-4b6c-b8f8-7192945a8de6","timestamp":"2024-02-02T22:21:52Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0df04d57-df6f-4fbd-94dc-2e317f80cdea] cannot determine cloud provider ","requestId":"0df04d57-df6f-4fbd-94dc-2e317f80cdea","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0df04d57-df6f-4fbd-94dc-2e317f80cdea] cannot determine cloud provider ","requestId":"0df04d57-df6f-4fbd-94dc-2e317f80cdea","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=91dbfdae-123b-448f-98fc-2012123fc404] cannot determine cloud provider ","requestId":"91dbfdae-123b-448f-98fc-2012123fc404","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=6b425d37-ecdb-485a-a6e3-1b9c1b147c46] cannot determine cloud provider ","requestId":"6b425d37-ecdb-485a-a6e3-1b9c1b147c46","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0df04d57-df6f-4fbd-94dc-2e317f80cdea] cannot determine cloud provider ","requestId":"0df04d57-df6f-4fbd-94dc-2e317f80cdea","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=6b425d37-ecdb-485a-a6e3-1b9c1b147c46] failed to start BackupWorkspace: required entitlements not present for this operation
","requestId":"6b425d37-ecdb-485a-a6e3-1b9c1b147c46","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=91dbfdae-123b-448f-98fc-2012123fc404] cannot determine cloud provider ","requestId":"91dbfdae-123b-448f-98fc-2012123fc404","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0df04d57-df6f-4fbd-94dc-2e317f80cdea] cannot determine cloud provider ","requestId":"0df04d57-df6f-4fbd-94dc-2e317f80cdea","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d] cannot determine cloud provider ","requestId":"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d] cannot determine cloud provider ","requestId":"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d] Pre-flight validation Category: 'FAILED', Name: 'Backup name uniqueness validation', Message: 'Backup name uniqueness validation failed'.","requestId":"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d] {"id":"962ff5cc-401e-4a79-814d-974df5455bc4","timestamp":1706912513415,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:f6836bf1-3c6d-4b73-94e3-7f7f34804489","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Backup name uniqueness validation\",\"detailedMessage\":\"Backup name uniqueness validation failed\"}],\"requestID\":\"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d\"}","requestId":"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d","tenantType":"PAAS"}}}","requestId":"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Backup name uniqueness validation","detailedMessage":"Backup name uniqueness validation failed"}],"requestID":"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d"}
","requestId":"6d3ecde2-3f9d-4bee-8f58-18bb3ea0db3d","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a7805472-4bef-4e3f-9cab-1c48c2ad2d77] cannot determine cloud provider ","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a7805472-4bef-4e3f-9cab-1c48c2ad2d77] cannot determine cloud provider ","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a7805472-4bef-4e3f-9cab-1c48c2ad2d77] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=a7805472-4bef-4e3f-9cab-1c48c2ad2d77] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provision:started status incompatible'.","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a7805472-4bef-4e3f-9cab-1c48c2ad2d77] {"id":"cd2091c7-af0a-4277-84eb-c43731c1e272","timestamp":1706912513505,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:12b81c0d-0f57-4574-9e80-5b130a828c40","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provision:started status incompatible\"}],\"requestID\":\"a7805472-4bef-4e3f-9cab-1c48c2ad2d77\"}","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","tenantType":"PAAS"}}}","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=a7805472-4bef-4e3f-9cab-1c48c2ad2d77] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provision:started status incompatible"}],"requestID":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77"}
","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=ad734a30-3b03-4b30-b617-67532157b198] cannot determine cloud provider ","requestId":"ad734a30-3b03-4b30-b617-67532157b198","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=ad734a30-3b03-4b30-b617-67532157b198] cannot determine cloud provider ","requestId":"ad734a30-3b03-4b30-b617-67532157b198","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=ad734a30-3b03-4b30-b617-67532157b198] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"ad734a30-3b03-4b30-b617-67532157b198","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=ad734a30-3b03-4b30-b617-67532157b198] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provision:finished status incompatible'.","requestId":"ad734a30-3b03-4b30-b617-67532157b198","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=ad734a30-3b03-4b30-b617-67532157b198] {"id":"9abd9c34-86bf-4f2b-a42a-63f7d586766a","timestamp":1706912513647,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3acb1409-0f4e-47e8-82ac-900b606c40a9","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provision:finished status incompatible\"}],\"requestID\":\"ad734a30-3b03-4b30-b617-67532157b198\"}","requestId":"ad734a30-3b03-4b30-b617-67532157b198","tenantType":"PAAS"}}}","requestId":"ad734a30-3b03-4b30-b617-67532157b198","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=ad734a30-3b03-4b30-b617-67532157b198] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provision:finished status incompatible"}],"requestID":"ad734a30-3b03-4b30-b617-67532157b198"}
","requestId":"ad734a30-3b03-4b30-b617-67532157b198","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_provision:failed_status[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=7442e16e-e5de-4e83-92a7-428d81366e1f] cannot determine cloud provider ","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=7442e16e-e5de-4e83-92a7-428d81366e1f] cannot determine cloud provider ","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=7442e16e-e5de-4e83-92a7-428d81366e1f] {"id":"f174813b-d2ad-455b-9474-4fa3a9fa29b8","timestamp":1706912513774,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:49b3ff5a-cb71-4647-8511-c356dc653e0c","backupName":"test-backup-provision-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","tenantType":"PAAS"}}}","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: [0m[33m[0.175ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-provision-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=7442e16e-e5de-4e83-92a7-428d81366e1f] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=7442e16e-e5de-4e83-92a7-428d81366e1f] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provision:failed status incompatible'.","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=7442e16e-e5de-4e83-92a7-428d81366e1f] {"id":"14a38fb3-351b-46e8-ad58-8daa04bcb4ce","timestamp":1706912513782,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:49b3ff5a-cb71-4647-8511-c356dc653e0c","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provision:failed status incompatible\"}],\"requestID\":\"7442e16e-e5de-4e83-92a7-428d81366e1f\"}","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","tenantType":"PAAS"}}}","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=7442e16e-e5de-4e83-92a7-428d81366e1f] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provision:failed status incompatible"}],"requestID":"7442e16e-e5de-4e83-92a7-428d81366e1f"}
","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=58ccfa95-d9d9-45e7-a3cf-a002dadf9b30] cannot determine cloud provider ","requestId":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=58ccfa95-d9d9-45e7-a3cf-a002dadf9b30] cannot determine cloud provider ","requestId":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=58ccfa95-d9d9-45e7-a3cf-a002dadf9b30] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=58ccfa95-d9d9-45e7-a3cf-a002dadf9b30] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provision:warning status incompatible'.","requestId":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=58ccfa95-d9d9-45e7-a3cf-a002dadf9b30] {"id":"f3270491-e81f-4c0b-b7ca-44a5b8bea75a","timestamp":1706912513919,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:86f52c78-4070-45d1-a0d0-31b7a172ac69","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provision:warning status incompatible\"}],\"requestID\":\"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30\"}","requestId":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30","tenantType":"PAAS"}}}","requestId":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=58ccfa95-d9d9-45e7-a3cf-a002dadf9b30] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provision:warning status incompatible"}],"requestID":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30"}
","requestId":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30","timestamp":"2024-02-02T22:21:53Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e388da10-71c0-456e-856d-a08e7c98a59a] cannot determine cloud provider ","requestId":"e388da10-71c0-456e-856d-a08e7c98a59a","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e388da10-71c0-456e-856d-a08e7c98a59a] cannot determine cloud provider ","requestId":"e388da10-71c0-456e-856d-a08e7c98a59a","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=e388da10-71c0-456e-856d-a08e7c98a59a] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"e388da10-71c0-456e-856d-a08e7c98a59a","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=e388da10-71c0-456e-856d-a08e7c98a59a] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provision:succeeded status incompatible'.","requestId":"e388da10-71c0-456e-856d-a08e7c98a59a","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=e388da10-71c0-456e-856d-a08e7c98a59a] {"id":"9bd2fee0-93f8-45d9-b7f5-8a50084d0cb9","timestamp":1706912514060,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d0f8968d-c19a-47e8-9ff8-efdabdabad0b","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provision:succeeded status incompatible\"}],\"requestID\":\"e388da10-71c0-456e-856d-a08e7c98a59a\"}","requestId":"e388da10-71c0-456e-856d-a08e7c98a59a","tenantType":"PAAS"}}}","requestId":"e388da10-71c0-456e-856d-a08e7c98a59a","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=e388da10-71c0-456e-856d-a08e7c98a59a] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provision:succeeded status incompatible"}],"requestID":"e388da10-71c0-456e-856d-a08e7c98a59a"}
","requestId":"e388da10-71c0-456e-856d-a08e7c98a59a","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a12d1fa1-7160-464d-825c-b8373a45a717] cannot determine cloud provider ","requestId":"a12d1fa1-7160-464d-825c-b8373a45a717","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a12d1fa1-7160-464d-825c-b8373a45a717] cannot determine cloud provider ","requestId":"a12d1fa1-7160-464d-825c-b8373a45a717","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a12d1fa1-7160-464d-825c-b8373a45a717] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a12d1fa1-7160-464d-825c-b8373a45a717","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=a12d1fa1-7160-464d-825c-b8373a45a717] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. deprovision:started status incompatible'.","requestId":"a12d1fa1-7160-464d-825c-b8373a45a717","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a12d1fa1-7160-464d-825c-b8373a45a717] {"id":"6e0edeb8-c8ec-4873-b4a3-cf891fedc571","timestamp":1706912514194,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:8c37cf79-20bf-4ac3-a6f7-e0c3e8512dd2","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. deprovision:started status incompatible\"}],\"requestID\":\"a12d1fa1-7160-464d-825c-b8373a45a717\"}","requestId":"a12d1fa1-7160-464d-825c-b8373a45a717","tenantType":"PAAS"}}}","requestId":"a12d1fa1-7160-464d-825c-b8373a45a717","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=a12d1fa1-7160-464d-825c-b8373a45a717] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. deprovision:started status incompatible"}],"requestID":"a12d1fa1-7160-464d-825c-b8373a45a717"}
","requestId":"a12d1fa1-7160-464d-825c-b8373a45a717","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=12e39425-a08b-487b-8455-e6d04393c419] cannot determine cloud provider ","requestId":"12e39425-a08b-487b-8455-e6d04393c419","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=12e39425-a08b-487b-8455-e6d04393c419] cannot determine cloud provider ","requestId":"12e39425-a08b-487b-8455-e6d04393c419","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=12e39425-a08b-487b-8455-e6d04393c419] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"12e39425-a08b-487b-8455-e6d04393c419","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=12e39425-a08b-487b-8455-e6d04393c419] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. deprovision:finished status incompatible'.","requestId":"12e39425-a08b-487b-8455-e6d04393c419","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=12e39425-a08b-487b-8455-e6d04393c419] {"id":"4216c2dd-db81-4627-9942-dbb5a5a5f37b","timestamp":1706912514329,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:25f67296-0b60-4166-bb60-242b5496c87b","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. deprovision:finished status incompatible\"}],\"requestID\":\"12e39425-a08b-487b-8455-e6d04393c419\"}","requestId":"12e39425-a08b-487b-8455-e6d04393c419","tenantType":"PAAS"}}}","requestId":"12e39425-a08b-487b-8455-e6d04393c419","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=12e39425-a08b-487b-8455-e6d04393c419] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. deprovision:finished status incompatible"}],"requestID":"12e39425-a08b-487b-8455-e6d04393c419"}
","requestId":"12e39425-a08b-487b-8455-e6d04393c419","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_deprovision:failed_status[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=df8129fb-a3f7-4571-bb1b-2baec88325fc] cannot determine cloud provider ","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=df8129fb-a3f7-4571-bb1b-2baec88325fc] cannot determine cloud provider ","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=df8129fb-a3f7-4571-bb1b-2baec88325fc] {"id":"d728460a-aec0-4fb2-98c0-02d706d1b8b8","timestamp":1706912514455,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:548690be-dd73-4f1a-9b23-bcb7e32942d2","backupName":"test-backup-deprovision-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","tenantType":"PAAS"}}}","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: [0m[33m[0.241ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-deprovision-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=df8129fb-a3f7-4571-bb1b-2baec88325fc] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=df8129fb-a3f7-4571-bb1b-2baec88325fc] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. deprovision:failed status incompatible'.","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=df8129fb-a3f7-4571-bb1b-2baec88325fc] {"id":"4a451efc-5ebc-4a47-85e3-5f8b1f6a013d","timestamp":1706912514465,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:548690be-dd73-4f1a-9b23-bcb7e32942d2","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. deprovision:failed status incompatible\"}],\"requestID\":\"df8129fb-a3f7-4571-bb1b-2baec88325fc\"}","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","tenantType":"PAAS"}}}","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=df8129fb-a3f7-4571-bb1b-2baec88325fc] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. deprovision:failed status incompatible"}],"requestID":"df8129fb-a3f7-4571-bb1b-2baec88325fc"}
","requestId":"df8129fb-a3f7-4571-bb1b-2baec88325fc","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=4a7d701d-df87-4366-b48e-7580aefb12db] cannot determine cloud provider ","requestId":"4a7d701d-df87-4366-b48e-7580aefb12db","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=4a7d701d-df87-4366-b48e-7580aefb12db] cannot determine cloud provider ","requestId":"4a7d701d-df87-4366-b48e-7580aefb12db","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=4a7d701d-df87-4366-b48e-7580aefb12db] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"4a7d701d-df87-4366-b48e-7580aefb12db","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=4a7d701d-df87-4366-b48e-7580aefb12db] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. deprovision:warning status incompatible'.","requestId":"4a7d701d-df87-4366-b48e-7580aefb12db","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=4a7d701d-df87-4366-b48e-7580aefb12db] {"id":"5fa1475d-804e-4664-84b0-aab6ce0e4bb5","timestamp":1706912514606,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:023e38a1-a4d7-421a-ba49-7a79ae568819","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. deprovision:warning status incompatible\"}],\"requestID\":\"4a7d701d-df87-4366-b48e-7580aefb12db\"}","requestId":"4a7d701d-df87-4366-b48e-7580aefb12db","tenantType":"PAAS"}}}","requestId":"4a7d701d-df87-4366-b48e-7580aefb12db","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=4a7d701d-df87-4366-b48e-7580aefb12db] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. deprovision:warning status incompatible"}],"requestID":"4a7d701d-df87-4366-b48e-7580aefb12db"}
","requestId":"4a7d701d-df87-4366-b48e-7580aefb12db","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=b53e72ed-a394-4797-99b9-09db944f4cf8] cannot determine cloud provider ","requestId":"b53e72ed-a394-4797-99b9-09db944f4cf8","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=b53e72ed-a394-4797-99b9-09db944f4cf8] cannot determine cloud provider ","requestId":"b53e72ed-a394-4797-99b9-09db944f4cf8","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=b53e72ed-a394-4797-99b9-09db944f4cf8] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"b53e72ed-a394-4797-99b9-09db944f4cf8","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=b53e72ed-a394-4797-99b9-09db944f4cf8] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. deprovision:succeeded status incompatible'.","requestId":"b53e72ed-a394-4797-99b9-09db944f4cf8","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=b53e72ed-a394-4797-99b9-09db944f4cf8] {"id":"c5424573-ef92-4243-b1d3-ed2bfda4613c","timestamp":1706912514745,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:e7ea1603-7d34-4492-ab1f-ed4cf4a8533b","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. deprovision:succeeded status incompatible\"}],\"requestID\":\"b53e72ed-a394-4797-99b9-09db944f4cf8\"}","requestId":"b53e72ed-a394-4797-99b9-09db944f4cf8","tenantType":"PAAS"}}}","requestId":"b53e72ed-a394-4797-99b9-09db944f4cf8","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=b53e72ed-a394-4797-99b9-09db944f4cf8] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. deprovision:succeeded status incompatible"}],"requestID":"b53e72ed-a394-4797-99b9-09db944f4cf8"}
","requestId":"b53e72ed-a394-4797-99b9-09db944f4cf8","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=9f4b12c2-2b36-45b2-a6b0-004206449df2] cannot determine cloud provider ","requestId":"9f4b12c2-2b36-45b2-a6b0-004206449df2","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=9f4b12c2-2b36-45b2-a6b0-004206449df2] cannot determine cloud provider ","requestId":"9f4b12c2-2b36-45b2-a6b0-004206449df2","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=9f4b12c2-2b36-45b2-a6b0-004206449df2] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"9f4b12c2-2b36-45b2-a6b0-004206449df2","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=9f4b12c2-2b36-45b2-a6b0-004206449df2] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. installation:started status incompatible'.","requestId":"9f4b12c2-2b36-45b2-a6b0-004206449df2","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=9f4b12c2-2b36-45b2-a6b0-004206449df2] {"id":"9fb2af45-8491-4dd5-804d-62b0b471c2c5","timestamp":1706912514888,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:cb1b607f-ddbd-4a64-8d56-59ba9f3da2d8","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. installation:started status incompatible\"}],\"requestID\":\"9f4b12c2-2b36-45b2-a6b0-004206449df2\"}","requestId":"9f4b12c2-2b36-45b2-a6b0-004206449df2","tenantType":"PAAS"}}}","requestId":"9f4b12c2-2b36-45b2-a6b0-004206449df2","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=9f4b12c2-2b36-45b2-a6b0-004206449df2] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. installation:started status incompatible"}],"requestID":"9f4b12c2-2b36-45b2-a6b0-004206449df2"}
","requestId":"9f4b12c2-2b36-45b2-a6b0-004206449df2","timestamp":"2024-02-02T22:21:54Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=f7373ee1-3090-4214-8214-73c94b5bf4ea] cannot determine cloud provider ","requestId":"f7373ee1-3090-4214-8214-73c94b5bf4ea","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=f7373ee1-3090-4214-8214-73c94b5bf4ea] cannot determine cloud provider ","requestId":"f7373ee1-3090-4214-8214-73c94b5bf4ea","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_installation:failed_status[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0ae9cab9-549a-4cf8-8562-0e963e3de883] cannot determine cloud provider ","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0ae9cab9-549a-4cf8-8562-0e963e3de883] cannot determine cloud provider ","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=0ae9cab9-549a-4cf8-8562-0e963e3de883] {"id":"1042b069-125f-4110-b368-a9beab9d42bf","timestamp":1706912515159,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:e179cef2-51fe-4a5e-b52a-59c6f9527daa","backupName":"test-backup-installation-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","tenantType":"PAAS"}}}","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=0ae9cab9-549a-4cf8-8562-0e963e3de883] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=0ae9cab9-549a-4cf8-8562-0e963e3de883] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. installation:failed status incompatible'.","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: [0m[33m[0.280ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-installation-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=0ae9cab9-549a-4cf8-8562-0e963e3de883] {"id":"de3066f2-1fd4-45be-b93d-a6aa8a3f25a1","timestamp":1706912515187,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:e179cef2-51fe-4a5e-b52a-59c6f9527daa","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. installation:failed status incompatible\"}],\"requestID\":\"0ae9cab9-549a-4cf8-8562-0e963e3de883\"}","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","tenantType":"PAAS"}}}","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=0ae9cab9-549a-4cf8-8562-0e963e3de883] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. installation:failed status incompatible"}],"requestID":"0ae9cab9-549a-4cf8-8562-0e963e3de883"}
","requestId":"0ae9cab9-549a-4cf8-8562-0e963e3de883","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d93c42bf-af6b-4805-bba0-7072fb7997c1] cannot determine cloud provider ","requestId":"d93c42bf-af6b-4805-bba0-7072fb7997c1","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d93c42bf-af6b-4805-bba0-7072fb7997c1] cannot determine cloud provider ","requestId":"d93c42bf-af6b-4805-bba0-7072fb7997c1","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=d93c42bf-af6b-4805-bba0-7072fb7997c1] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"d93c42bf-af6b-4805-bba0-7072fb7997c1","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=d93c42bf-af6b-4805-bba0-7072fb7997c1] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. installation:warning status incompatible'.","requestId":"d93c42bf-af6b-4805-bba0-7072fb7997c1","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=d93c42bf-af6b-4805-bba0-7072fb7997c1] {"id":"89081979-2cc1-4e25-91a0-788be1a0d20f","timestamp":1706912515335,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:6ed4c43b-001a-4630-9290-f7cbe618ac24","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. installation:warning status incompatible\"}],\"requestID\":\"d93c42bf-af6b-4805-bba0-7072fb7997c1\"}","requestId":"d93c42bf-af6b-4805-bba0-7072fb7997c1","tenantType":"PAAS"}}}","requestId":"d93c42bf-af6b-4805-bba0-7072fb7997c1","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=d93c42bf-af6b-4805-bba0-7072fb7997c1] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. installation:warning status incompatible"}],"requestID":"d93c42bf-af6b-4805-bba0-7072fb7997c1"}
","requestId":"d93c42bf-af6b-4805-bba0-7072fb7997c1","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=abe713c8-508b-470f-ad00-2a681e5d8359] cannot determine cloud provider ","requestId":"abe713c8-508b-470f-ad00-2a681e5d8359","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=abe713c8-508b-470f-ad00-2a681e5d8359] cannot determine cloud provider ","requestId":"abe713c8-508b-470f-ad00-2a681e5d8359","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=abe713c8-508b-470f-ad00-2a681e5d8359] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"abe713c8-508b-470f-ad00-2a681e5d8359","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=abe713c8-508b-470f-ad00-2a681e5d8359] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. installation:succeeded status incompatible'.","requestId":"abe713c8-508b-470f-ad00-2a681e5d8359","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=abe713c8-508b-470f-ad00-2a681e5d8359] {"id":"03c92209-e56d-481f-bf99-df6272348e6a","timestamp":1706912515472,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:14403ab3-0b38-41f7-b447-a4cec0c44b55","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. installation:succeeded status incompatible\"}],\"requestID\":\"abe713c8-508b-470f-ad00-2a681e5d8359\"}","requestId":"abe713c8-508b-470f-ad00-2a681e5d8359","tenantType":"PAAS"}}}","requestId":"abe713c8-508b-470f-ad00-2a681e5d8359","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=abe713c8-508b-470f-ad00-2a681e5d8359] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. installation:succeeded status incompatible"}],"requestID":"abe713c8-508b-470f-ad00-2a681e5d8359"}
","requestId":"abe713c8-508b-470f-ad00-2a681e5d8359","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=84fdde98-878f-462c-b0fb-aabb1816ffa2] cannot determine cloud provider ","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=84fdde98-878f-462c-b0fb-aabb1816ffa2] cannot determine cloud provider ","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=84fdde98-878f-462c-b0fb-aabb1816ffa2] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=84fdde98-878f-462c-b0fb-aabb1816ffa2] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. mount:started status incompatible'.","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=84fdde98-878f-462c-b0fb-aabb1816ffa2] {"id":"75df3b88-3431-45b6-a462-4eb6cf43dcde","timestamp":1706912515615,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:c108298b-211b-4907-afa1-2ea48019ecae","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. mount:started status incompatible\"}],\"requestID\":\"84fdde98-878f-462c-b0fb-aabb1816ffa2\"}","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","tenantType":"PAAS"}}}","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=84fdde98-878f-462c-b0fb-aabb1816ffa2] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. mount:started status incompatible"}],"requestID":"84fdde98-878f-462c-b0fb-aabb1816ffa2"}
","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=bf350beb-b634-4417-a0ed-82a717849407] cannot determine cloud provider ","requestId":"bf350beb-b634-4417-a0ed-82a717849407","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=bf350beb-b634-4417-a0ed-82a717849407] cannot determine cloud provider ","requestId":"bf350beb-b634-4417-a0ed-82a717849407","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=bf350beb-b634-4417-a0ed-82a717849407] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"bf350beb-b634-4417-a0ed-82a717849407","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=bf350beb-b634-4417-a0ed-82a717849407] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. mount:finished status incompatible'.","requestId":"bf350beb-b634-4417-a0ed-82a717849407","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=bf350beb-b634-4417-a0ed-82a717849407] {"id":"7e46da5f-6ab1-486d-85a3-be26d9a8cd2e","timestamp":1706912515758,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3ae83ac9-6fe3-461f-b74d-1219b7d5c746","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. mount:finished status incompatible\"}],\"requestID\":\"bf350beb-b634-4417-a0ed-82a717849407\"}","requestId":"bf350beb-b634-4417-a0ed-82a717849407","tenantType":"PAAS"}}}","requestId":"bf350beb-b634-4417-a0ed-82a717849407","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=bf350beb-b634-4417-a0ed-82a717849407] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. mount:finished status incompatible"}],"requestID":"bf350beb-b634-4417-a0ed-82a717849407"}
","requestId":"bf350beb-b634-4417-a0ed-82a717849407","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_mount:failed_status[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=17789260-2e7a-4916-be5c-a72e5425627b] cannot determine cloud provider ","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:20 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=17789260-2e7a-4916-be5c-a72e5425627b] cannot determine cloud provider ","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=17789260-2e7a-4916-be5c-a72e5425627b] {"id":"b018fe77-7cb9-4468-bf01-5bdcb6258e79","timestamp":1706912515887,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:0aec2618-357d-4b6b-ab3c-ab6a9da56017","backupName":"test-backup-mount-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","tenantType":"PAAS"}}}","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: [0m[33m[0.245ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-mount-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=17789260-2e7a-4916-be5c-a72e5425627b] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=17789260-2e7a-4916-be5c-a72e5425627b] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. mount:failed status incompatible'.","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=17789260-2e7a-4916-be5c-a72e5425627b] {"id":"456b1bb4-649d-4de1-8b7c-4bcc4b6a933c","timestamp":1706912515896,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:0aec2618-357d-4b6b-ab3c-ab6a9da56017","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. mount:failed status incompatible\"}],\"requestID\":\"17789260-2e7a-4916-be5c-a72e5425627b\"}","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","tenantType":"PAAS"}}}","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=17789260-2e7a-4916-be5c-a72e5425627b] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. mount:failed status incompatible"}],"requestID":"17789260-2e7a-4916-be5c-a72e5425627b"}
","requestId":"17789260-2e7a-4916-be5c-a72e5425627b","timestamp":"2024-02-02T22:21:55Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=174d5107-dc06-4d6a-b741-4a4d133966da] cannot determine cloud provider ","requestId":"174d5107-dc06-4d6a-b741-4a4d133966da","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=174d5107-dc06-4d6a-b741-4a4d133966da] cannot determine cloud provider ","requestId":"174d5107-dc06-4d6a-b741-4a4d133966da","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=174d5107-dc06-4d6a-b741-4a4d133966da] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"174d5107-dc06-4d6a-b741-4a4d133966da","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=174d5107-dc06-4d6a-b741-4a4d133966da] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. mount:warning status incompatible'.","requestId":"174d5107-dc06-4d6a-b741-4a4d133966da","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=174d5107-dc06-4d6a-b741-4a4d133966da] {"id":"555451c9-486f-4e50-ab25-ce0ae257d66b","timestamp":1706912516032,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:e63feea5-6568-4d11-a227-c8b919062983","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. mount:warning status incompatible\"}],\"requestID\":\"174d5107-dc06-4d6a-b741-4a4d133966da\"}","requestId":"174d5107-dc06-4d6a-b741-4a4d133966da","tenantType":"PAAS"}}}","requestId":"174d5107-dc06-4d6a-b741-4a4d133966da","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=174d5107-dc06-4d6a-b741-4a4d133966da] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. mount:warning status incompatible"}],"requestID":"174d5107-dc06-4d6a-b741-4a4d133966da"}
","requestId":"174d5107-dc06-4d6a-b741-4a4d133966da","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=9b3c9154-b7bf-4a3d-81b2-4ab635e35542] cannot determine cloud provider ","requestId":"9b3c9154-b7bf-4a3d-81b2-4ab635e35542","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=9b3c9154-b7bf-4a3d-81b2-4ab635e35542] cannot determine cloud provider ","requestId":"9b3c9154-b7bf-4a3d-81b2-4ab635e35542","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=9b3c9154-b7bf-4a3d-81b2-4ab635e35542] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"9b3c9154-b7bf-4a3d-81b2-4ab635e35542","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=9b3c9154-b7bf-4a3d-81b2-4ab635e35542] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. mount:succeeded status incompatible'.","requestId":"9b3c9154-b7bf-4a3d-81b2-4ab635e35542","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=9b3c9154-b7bf-4a3d-81b2-4ab635e35542] {"id":"1d9b9183-1746-43d1-ae83-82031169ae76","timestamp":1706912516182,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:6b7e753d-ddff-4321-ae54-18f1213cffff","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. mount:succeeded status incompatible\"}],\"requestID\":\"9b3c9154-b7bf-4a3d-81b2-4ab635e35542\"}","requestId":"9b3c9154-b7bf-4a3d-81b2-4ab635e35542","tenantType":"PAAS"}}}","requestId":"9b3c9154-b7bf-4a3d-81b2-4ab635e35542","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=9b3c9154-b7bf-4a3d-81b2-4ab635e35542] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. mount:succeeded status incompatible"}],"requestID":"9b3c9154-b7bf-4a3d-81b2-4ab635e35542"}
","requestId":"9b3c9154-b7bf-4a3d-81b2-4ab635e35542","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0248b81f-c441-4538-a34c-2729602ccbdb] cannot determine cloud provider ","requestId":"0248b81f-c441-4538-a34c-2729602ccbdb","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0248b81f-c441-4538-a34c-2729602ccbdb] cannot determine cloud provider ","requestId":"0248b81f-c441-4538-a34c-2729602ccbdb","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=0248b81f-c441-4538-a34c-2729602ccbdb] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"0248b81f-c441-4538-a34c-2729602ccbdb","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=0248b81f-c441-4538-a34c-2729602ccbdb] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. renew:started status incompatible'.","requestId":"0248b81f-c441-4538-a34c-2729602ccbdb","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=0248b81f-c441-4538-a34c-2729602ccbdb] {"id":"7d02ab58-9510-4bbe-96c8-6c9eb38d5529","timestamp":1706912516322,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3af83589-5d90-40e7-b63f-77513f0cc1a9","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. renew:started status incompatible\"}],\"requestID\":\"0248b81f-c441-4538-a34c-2729602ccbdb\"}","requestId":"0248b81f-c441-4538-a34c-2729602ccbdb","tenantType":"PAAS"}}}","requestId":"0248b81f-c441-4538-a34c-2729602ccbdb","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=0248b81f-c441-4538-a34c-2729602ccbdb] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. renew:started status incompatible"}],"requestID":"0248b81f-c441-4538-a34c-2729602ccbdb"}
","requestId":"0248b81f-c441-4538-a34c-2729602ccbdb","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=5931bf62-395b-4453-bb2d-d1d96c51090c] cannot determine cloud provider ","requestId":"5931bf62-395b-4453-bb2d-d1d96c51090c","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=5931bf62-395b-4453-bb2d-d1d96c51090c] cannot determine cloud provider ","requestId":"5931bf62-395b-4453-bb2d-d1d96c51090c","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_renew:failed_status[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=20b8a987-277d-4c58-a1db-b3cc616acf11] cannot determine cloud provider ","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=20b8a987-277d-4c58-a1db-b3cc616acf11] cannot determine cloud provider ","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=20b8a987-277d-4c58-a1db-b3cc616acf11] {"id":"48e5ea17-e920-4e0a-9cf2-8252583f53cc","timestamp":1706912516602,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:b5d10e21-9984-4643-b792-41a2159c9189","backupName":"test-backup-renew-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","tenantType":"PAAS"}}}","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: [0m[33m[0.236ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-renew-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=20b8a987-277d-4c58-a1db-b3cc616acf11] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=20b8a987-277d-4c58-a1db-b3cc616acf11] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. renew:failed status incompatible'.","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=20b8a987-277d-4c58-a1db-b3cc616acf11] {"id":"b106d96a-7992-4473-a36a-37fec15e3764","timestamp":1706912516614,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:b5d10e21-9984-4643-b792-41a2159c9189","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. renew:failed status incompatible\"}],\"requestID\":\"20b8a987-277d-4c58-a1db-b3cc616acf11\"}","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","tenantType":"PAAS"}}}","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=20b8a987-277d-4c58-a1db-b3cc616acf11] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. renew:failed status incompatible"}],"requestID":"20b8a987-277d-4c58-a1db-b3cc616acf11"}
","requestId":"20b8a987-277d-4c58-a1db-b3cc616acf11","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=58482f3a-6b53-4bc9-a79c-c4b408a737ac] cannot determine cloud provider ","requestId":"58482f3a-6b53-4bc9-a79c-c4b408a737ac","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=58482f3a-6b53-4bc9-a79c-c4b408a737ac] cannot determine cloud provider ","requestId":"58482f3a-6b53-4bc9-a79c-c4b408a737ac","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=58482f3a-6b53-4bc9-a79c-c4b408a737ac] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"58482f3a-6b53-4bc9-a79c-c4b408a737ac","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=58482f3a-6b53-4bc9-a79c-c4b408a737ac] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. renew:warning status incompatible'.","requestId":"58482f3a-6b53-4bc9-a79c-c4b408a737ac","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=58482f3a-6b53-4bc9-a79c-c4b408a737ac] {"id":"641d0577-d4fd-4a7c-8676-3f9e7f0bc9b0","timestamp":1706912516752,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:2e492e92-e9ad-46f6-a5b8-9cd7448c92c3","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. renew:warning status incompatible\"}],\"requestID\":\"58482f3a-6b53-4bc9-a79c-c4b408a737ac\"}","requestId":"58482f3a-6b53-4bc9-a79c-c4b408a737ac","tenantType":"PAAS"}}}","requestId":"58482f3a-6b53-4bc9-a79c-c4b408a737ac","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=58482f3a-6b53-4bc9-a79c-c4b408a737ac] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. renew:warning status incompatible"}],"requestID":"58482f3a-6b53-4bc9-a79c-c4b408a737ac"}
","requestId":"58482f3a-6b53-4bc9-a79c-c4b408a737ac","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=9a741e9f-93a8-408d-89e9-6d96cddc91d0] cannot determine cloud provider ","requestId":"9a741e9f-93a8-408d-89e9-6d96cddc91d0","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=9a741e9f-93a8-408d-89e9-6d96cddc91d0] cannot determine cloud provider ","requestId":"9a741e9f-93a8-408d-89e9-6d96cddc91d0","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=9a741e9f-93a8-408d-89e9-6d96cddc91d0] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"9a741e9f-93a8-408d-89e9-6d96cddc91d0","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=9a741e9f-93a8-408d-89e9-6d96cddc91d0] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. renew:succeeded status incompatible'.","requestId":"9a741e9f-93a8-408d-89e9-6d96cddc91d0","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=9a741e9f-93a8-408d-89e9-6d96cddc91d0] {"id":"ce910cde-d999-4336-9a9c-88947cf7c6bf","timestamp":1706912516894,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:cba78955-c27e-48c3-a33d-d886d485bc69","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. renew:succeeded status incompatible\"}],\"requestID\":\"9a741e9f-93a8-408d-89e9-6d96cddc91d0\"}","requestId":"9a741e9f-93a8-408d-89e9-6d96cddc91d0","tenantType":"PAAS"}}}","requestId":"9a741e9f-93a8-408d-89e9-6d96cddc91d0","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=9a741e9f-93a8-408d-89e9-6d96cddc91d0] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. renew:succeeded status incompatible"}],"requestID":"9a741e9f-93a8-408d-89e9-6d96cddc91d0"}
","requestId":"9a741e9f-93a8-408d-89e9-6d96cddc91d0","timestamp":"2024-02-02T22:21:56Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d1585484-aef7-4cfd-b40f-36a3492d8e50] cannot determine cloud provider ","requestId":"d1585484-aef7-4cfd-b40f-36a3492d8e50","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d1585484-aef7-4cfd-b40f-36a3492d8e50] cannot determine cloud provider ","requestId":"d1585484-aef7-4cfd-b40f-36a3492d8e50","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=d1585484-aef7-4cfd-b40f-36a3492d8e50] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"d1585484-aef7-4cfd-b40f-36a3492d8e50","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=d1585484-aef7-4cfd-b40f-36a3492d8e50] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. suspend:started status incompatible'.","requestId":"d1585484-aef7-4cfd-b40f-36a3492d8e50","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=d1585484-aef7-4cfd-b40f-36a3492d8e50] {"id":"320a7807-ebfa-475c-83fe-82d32f02f451","timestamp":1706912517043,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:eb78ca77-3880-4079-b938-994f218c6111","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. suspend:started status incompatible\"}],\"requestID\":\"d1585484-aef7-4cfd-b40f-36a3492d8e50\"}","requestId":"d1585484-aef7-4cfd-b40f-36a3492d8e50","tenantType":"PAAS"}}}","requestId":"d1585484-aef7-4cfd-b40f-36a3492d8e50","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=d1585484-aef7-4cfd-b40f-36a3492d8e50] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. suspend:started status incompatible"}],"requestID":"d1585484-aef7-4cfd-b40f-36a3492d8e50"}
","requestId":"d1585484-aef7-4cfd-b40f-36a3492d8e50","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e039f869-1b6b-46a8-81c6-bc9eb67dce58] cannot determine cloud provider ","requestId":"e039f869-1b6b-46a8-81c6-bc9eb67dce58","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e039f869-1b6b-46a8-81c6-bc9eb67dce58] cannot determine cloud provider ","requestId":"e039f869-1b6b-46a8-81c6-bc9eb67dce58","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_suspend:failed_status[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=15994f4b-75de-454b-a27a-df74c38184d4] cannot determine cloud provider ","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=15994f4b-75de-454b-a27a-df74c38184d4] cannot determine cloud provider ","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=15994f4b-75de-454b-a27a-df74c38184d4] {"id":"2f20b8aa-a50c-4146-aa54-69b53c535348","timestamp":1706912517312,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d052e0b7-77d9-4674-b231-84bc3e459712","backupName":"test-backup-suspend-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","tenantType":"PAAS"}}}","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=15994f4b-75de-454b-a27a-df74c38184d4] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=15994f4b-75de-454b-a27a-df74c38184d4] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. suspend:failed status incompatible'.","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: [0m[33m[0.245ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-suspend-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=15994f4b-75de-454b-a27a-df74c38184d4] {"id":"f9012ad9-7eb1-4ea5-9758-30e7b4f191b9","timestamp":1706912517321,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d052e0b7-77d9-4674-b231-84bc3e459712","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. suspend:failed status incompatible\"}],\"requestID\":\"15994f4b-75de-454b-a27a-df74c38184d4\"}","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","tenantType":"PAAS"}}}","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=15994f4b-75de-454b-a27a-df74c38184d4] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. suspend:failed status incompatible"}],"requestID":"15994f4b-75de-454b-a27a-df74c38184d4"}
","requestId":"15994f4b-75de-454b-a27a-df74c38184d4","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=71ed0637-6e94-42b2-9436-cb016362fe6f] cannot determine cloud provider ","requestId":"71ed0637-6e94-42b2-9436-cb016362fe6f","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=71ed0637-6e94-42b2-9436-cb016362fe6f] cannot determine cloud provider ","requestId":"71ed0637-6e94-42b2-9436-cb016362fe6f","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=71ed0637-6e94-42b2-9436-cb016362fe6f] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"71ed0637-6e94-42b2-9436-cb016362fe6f","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=71ed0637-6e94-42b2-9436-cb016362fe6f] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. suspend:warning status incompatible'.","requestId":"71ed0637-6e94-42b2-9436-cb016362fe6f","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=71ed0637-6e94-42b2-9436-cb016362fe6f] {"id":"4c1c895c-af2a-4ee0-9de4-1c1536f91ebf","timestamp":1706912517452,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:2d1e2d1b-b125-4fb2-b136-23d848753c2e","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. suspend:warning status incompatible\"}],\"requestID\":\"71ed0637-6e94-42b2-9436-cb016362fe6f\"}","requestId":"71ed0637-6e94-42b2-9436-cb016362fe6f","tenantType":"PAAS"}}}","requestId":"71ed0637-6e94-42b2-9436-cb016362fe6f","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=71ed0637-6e94-42b2-9436-cb016362fe6f] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. suspend:warning status incompatible"}],"requestID":"71ed0637-6e94-42b2-9436-cb016362fe6f"}
","requestId":"71ed0637-6e94-42b2-9436-cb016362fe6f","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a2930017-79d9-484a-9e72-4d0354563b58] cannot determine cloud provider ","requestId":"a2930017-79d9-484a-9e72-4d0354563b58","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a2930017-79d9-484a-9e72-4d0354563b58] cannot determine cloud provider ","requestId":"a2930017-79d9-484a-9e72-4d0354563b58","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a2930017-79d9-484a-9e72-4d0354563b58] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a2930017-79d9-484a-9e72-4d0354563b58","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=a2930017-79d9-484a-9e72-4d0354563b58] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. suspend:succeeded status incompatible'.","requestId":"a2930017-79d9-484a-9e72-4d0354563b58","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a2930017-79d9-484a-9e72-4d0354563b58] {"id":"2e397e2a-8ab2-4125-a043-b1d54e4ee2db","timestamp":1706912517591,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:4df0a061-0f91-48df-974e-c8d63a5e9c0d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. suspend:succeeded status incompatible\"}],\"requestID\":\"a2930017-79d9-484a-9e72-4d0354563b58\"}","requestId":"a2930017-79d9-484a-9e72-4d0354563b58","tenantType":"PAAS"}}}","requestId":"a2930017-79d9-484a-9e72-4d0354563b58","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=a2930017-79d9-484a-9e72-4d0354563b58] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. suspend:succeeded status incompatible"}],"requestID":"a2930017-79d9-484a-9e72-4d0354563b58"}
","requestId":"a2930017-79d9-484a-9e72-4d0354563b58","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=efe855ed-837d-4a08-941f-e727a64aeed1] cannot determine cloud provider ","requestId":"efe855ed-837d-4a08-941f-e727a64aeed1","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=efe855ed-837d-4a08-941f-e727a64aeed1] cannot determine cloud provider ","requestId":"efe855ed-837d-4a08-941f-e727a64aeed1","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=efe855ed-837d-4a08-941f-e727a64aeed1] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"efe855ed-837d-4a08-941f-e727a64aeed1","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=efe855ed-837d-4a08-941f-e727a64aeed1] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. resume:started status incompatible'.","requestId":"efe855ed-837d-4a08-941f-e727a64aeed1","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=efe855ed-837d-4a08-941f-e727a64aeed1] {"id":"41a4e8c3-b5a9-4e1c-ae40-71866dd72965","timestamp":1706912517725,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:9ded609b-e0d6-40c9-aff6-dffac9032e31","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. resume:started status incompatible\"}],\"requestID\":\"efe855ed-837d-4a08-941f-e727a64aeed1\"}","requestId":"efe855ed-837d-4a08-941f-e727a64aeed1","tenantType":"PAAS"}}}","requestId":"efe855ed-837d-4a08-941f-e727a64aeed1","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=efe855ed-837d-4a08-941f-e727a64aeed1] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. resume:started status incompatible"}],"requestID":"efe855ed-837d-4a08-941f-e727a64aeed1"}
","requestId":"efe855ed-837d-4a08-941f-e727a64aeed1","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=df53bbc4-31d6-4dc4-b85c-bb79e6775fd6] cannot determine cloud provider ","requestId":"df53bbc4-31d6-4dc4-b85c-bb79e6775fd6","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=df53bbc4-31d6-4dc4-b85c-bb79e6775fd6] cannot determine cloud provider ","requestId":"df53bbc4-31d6-4dc4-b85c-bb79e6775fd6","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_resume:failed_status[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=ce36e695-0cea-4e6d-b129-8eed316e7463] cannot determine cloud provider ","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","timestamp":"2024-02-02T22:21:57Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=ce36e695-0cea-4e6d-b129-8eed316e7463] cannot determine cloud provider ","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=ce36e695-0cea-4e6d-b129-8eed316e7463] {"id":"4bf4e5bc-5951-41b4-88fd-d2974f0aa7d4","timestamp":1706912518001,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:8817aa43-1cca-4205-8d5c-758c719d905a","backupName":"test-backup-resume-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","tenantType":"PAAS"}}}","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: [0m[33m[0.270ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-resume-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=ce36e695-0cea-4e6d-b129-8eed316e7463] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=ce36e695-0cea-4e6d-b129-8eed316e7463] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. resume:failed status incompatible'.","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=ce36e695-0cea-4e6d-b129-8eed316e7463] {"id":"487c78a6-b3a3-4612-b176-5720030a059e","timestamp":1706912518012,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:8817aa43-1cca-4205-8d5c-758c719d905a","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. resume:failed status incompatible\"}],\"requestID\":\"ce36e695-0cea-4e6d-b129-8eed316e7463\"}","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","tenantType":"PAAS"}}}","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=ce36e695-0cea-4e6d-b129-8eed316e7463] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. resume:failed status incompatible"}],"requestID":"ce36e695-0cea-4e6d-b129-8eed316e7463"}
","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d1932df7-1542-4f46-8648-ae615041d420] cannot determine cloud provider ","requestId":"d1932df7-1542-4f46-8648-ae615041d420","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d1932df7-1542-4f46-8648-ae615041d420] cannot determine cloud provider ","requestId":"d1932df7-1542-4f46-8648-ae615041d420","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=d1932df7-1542-4f46-8648-ae615041d420] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"d1932df7-1542-4f46-8648-ae615041d420","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=d1932df7-1542-4f46-8648-ae615041d420] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. resume:warning status incompatible'.","requestId":"d1932df7-1542-4f46-8648-ae615041d420","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=d1932df7-1542-4f46-8648-ae615041d420] {"id":"39702291-66a3-4452-84aa-97d29f764f16","timestamp":1706912518155,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:9f530103-a1f1-4bd5-bd31-8957810b20f3","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. resume:warning status incompatible\"}],\"requestID\":\"d1932df7-1542-4f46-8648-ae615041d420\"}","requestId":"d1932df7-1542-4f46-8648-ae615041d420","tenantType":"PAAS"}}}","requestId":"d1932df7-1542-4f46-8648-ae615041d420","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=d1932df7-1542-4f46-8648-ae615041d420] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. resume:warning status incompatible"}],"requestID":"d1932df7-1542-4f46-8648-ae615041d420"}
","requestId":"d1932df7-1542-4f46-8648-ae615041d420","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=563afb7f-02f0-444f-a1c1-20f05225c6a0] cannot determine cloud provider ","requestId":"563afb7f-02f0-444f-a1c1-20f05225c6a0","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=563afb7f-02f0-444f-a1c1-20f05225c6a0] cannot determine cloud provider ","requestId":"563afb7f-02f0-444f-a1c1-20f05225c6a0","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=563afb7f-02f0-444f-a1c1-20f05225c6a0] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"563afb7f-02f0-444f-a1c1-20f05225c6a0","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=563afb7f-02f0-444f-a1c1-20f05225c6a0] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. resume:succeeded status incompatible'.","requestId":"563afb7f-02f0-444f-a1c1-20f05225c6a0","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=563afb7f-02f0-444f-a1c1-20f05225c6a0] {"id":"ceab698e-9e26-4887-99c4-6e6613bb75b4","timestamp":1706912518300,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:a7c96efb-105c-4d4f-81c2-32c50f9e3d07","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. resume:succeeded status incompatible\"}],\"requestID\":\"563afb7f-02f0-444f-a1c1-20f05225c6a0\"}","requestId":"563afb7f-02f0-444f-a1c1-20f05225c6a0","tenantType":"PAAS"}}}","requestId":"563afb7f-02f0-444f-a1c1-20f05225c6a0","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=563afb7f-02f0-444f-a1c1-20f05225c6a0] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. resume:succeeded status incompatible"}],"requestID":"563afb7f-02f0-444f-a1c1-20f05225c6a0"}
","requestId":"563afb7f-02f0-444f-a1c1-20f05225c6a0","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d9da3741-efc3-4f86-a18d-8a2f0ede3e18] cannot determine cloud provider ","requestId":"d9da3741-efc3-4f86-a18d-8a2f0ede3e18","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d9da3741-efc3-4f86-a18d-8a2f0ede3e18] cannot determine cloud provider ","requestId":"d9da3741-efc3-4f86-a18d-8a2f0ede3e18","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=d9da3741-efc3-4f86-a18d-8a2f0ede3e18] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"d9da3741-efc3-4f86-a18d-8a2f0ede3e18","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=d9da3741-efc3-4f86-a18d-8a2f0ede3e18] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. upgrade:started status incompatible'.","requestId":"d9da3741-efc3-4f86-a18d-8a2f0ede3e18","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=d9da3741-efc3-4f86-a18d-8a2f0ede3e18] {"id":"bad7cf93-f38a-435f-8641-123a87126e7f","timestamp":1706912518442,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:9a313d4b-5dcc-4e8f-bef6-1ecf7627bc6c","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. upgrade:started status incompatible\"}],\"requestID\":\"d9da3741-efc3-4f86-a18d-8a2f0ede3e18\"}","requestId":"d9da3741-efc3-4f86-a18d-8a2f0ede3e18","tenantType":"PAAS"}}}","requestId":"d9da3741-efc3-4f86-a18d-8a2f0ede3e18","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=d9da3741-efc3-4f86-a18d-8a2f0ede3e18] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. upgrade:started status incompatible"}],"requestID":"d9da3741-efc3-4f86-a18d-8a2f0ede3e18"}
","requestId":"d9da3741-efc3-4f86-a18d-8a2f0ede3e18","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=aa5ef07a-4578-4b97-a25c-2c40678ed800] cannot determine cloud provider ","requestId":"aa5ef07a-4578-4b97-a25c-2c40678ed800","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=aa5ef07a-4578-4b97-a25c-2c40678ed800] cannot determine cloud provider ","requestId":"aa5ef07a-4578-4b97-a25c-2c40678ed800","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_upgrade:failed_status[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8cc4ec92-cf75-40b3-a27e-1636157fb240] cannot determine cloud provider ","requestId":"8cc4ec92-cf75-40b3-a27e-1636157fb240","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8cc4ec92-cf75-40b3-a27e-1636157fb240] cannot determine cloud provider ","requestId":"8cc4ec92-cf75-40b3-a27e-1636157fb240","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=8cc4ec92-cf75-40b3-a27e-1636157fb240] {"id":"dff39130-7ecc-40ff-ae73-4a5637b2e69c","timestamp":1706912518708,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:7bdfe5f1-4a4a-45e5-80dd-ec7592493613","backupName":"test-backup-upgrade-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"8cc4ec92-cf75-40b3-a27e-1636157fb240","tenantType":"PAAS"}}}","requestId":"8cc4ec92-cf75-40b3-a27e-1636157fb240","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: [0m[33m[0.242ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-upgrade-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=18c767af-938c-4e2c-903a-f15d3109d786] cannot determine cloud provider ","requestId":"18c767af-938c-4e2c-903a-f15d3109d786","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=18c767af-938c-4e2c-903a-f15d3109d786] cannot determine cloud provider ","requestId":"18c767af-938c-4e2c-903a-f15d3109d786","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=18c767af-938c-4e2c-903a-f15d3109d786] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"18c767af-938c-4e2c-903a-f15d3109d786","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=18c767af-938c-4e2c-903a-f15d3109d786] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. upgrade:warning status incompatible'.","requestId":"18c767af-938c-4e2c-903a-f15d3109d786","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=18c767af-938c-4e2c-903a-f15d3109d786] {"id":"7dda8e28-7923-4ebf-84d6-fdd483a242f1","timestamp":1706912518858,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:6d47da10-359a-4fc7-be90-002b87a03f87","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. upgrade:warning status incompatible\"}],\"requestID\":\"18c767af-938c-4e2c-903a-f15d3109d786\"}","requestId":"18c767af-938c-4e2c-903a-f15d3109d786","tenantType":"PAAS"}}}","requestId":"18c767af-938c-4e2c-903a-f15d3109d786","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=18c767af-938c-4e2c-903a-f15d3109d786] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. upgrade:warning status incompatible"}],"requestID":"18c767af-938c-4e2c-903a-f15d3109d786"}
","requestId":"18c767af-938c-4e2c-903a-f15d3109d786","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=64f0c13b-8136-4a2b-a774-4ae5c0d06c1c] cannot determine cloud provider ","requestId":"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=64f0c13b-8136-4a2b-a774-4ae5c0d06c1c] cannot determine cloud provider ","requestId":"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=64f0c13b-8136-4a2b-a774-4ae5c0d06c1c] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=64f0c13b-8136-4a2b-a774-4ae5c0d06c1c] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. upgrade:succeeded status incompatible'.","requestId":"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=64f0c13b-8136-4a2b-a774-4ae5c0d06c1c] {"id":"70081f67-e4ce-441d-8e48-b1277480ba95","timestamp":1706912518996,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:fd1b12e5-41b8-4059-9246-1ca79205d656","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. upgrade:succeeded status incompatible\"}],\"requestID\":\"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c\"}","requestId":"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c","tenantType":"PAAS"}}}","requestId":"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=64f0c13b-8136-4a2b-a774-4ae5c0d06c1c] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. upgrade:succeeded status incompatible"}],"requestID":"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c"}
","requestId":"64f0c13b-8136-4a2b-a774-4ae5c0d06c1c","timestamp":"2024-02-02T22:21:58Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e5b62bdf-2c50-408e-8a14-04f11f1b926b] cannot determine cloud provider ","requestId":"e5b62bdf-2c50-408e-8a14-04f11f1b926b","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e5b62bdf-2c50-408e-8a14-04f11f1b926b] cannot determine cloud provider ","requestId":"e5b62bdf-2c50-408e-8a14-04f11f1b926b","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=e5b62bdf-2c50-408e-8a14-04f11f1b926b] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"e5b62bdf-2c50-408e-8a14-04f11f1b926b","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=e5b62bdf-2c50-408e-8a14-04f11f1b926b] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. modify:started status incompatible'.","requestId":"e5b62bdf-2c50-408e-8a14-04f11f1b926b","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=e5b62bdf-2c50-408e-8a14-04f11f1b926b] {"id":"fcc47d61-2da8-4e32-8458-a47c26ba3a90","timestamp":1706912519141,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:551d46d8-1ea6-4449-8c34-705adaccd2b2","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. modify:started status incompatible\"}],\"requestID\":\"e5b62bdf-2c50-408e-8a14-04f11f1b926b\"}","requestId":"e5b62bdf-2c50-408e-8a14-04f11f1b926b","tenantType":"PAAS"}}}","requestId":"e5b62bdf-2c50-408e-8a14-04f11f1b926b","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=e5b62bdf-2c50-408e-8a14-04f11f1b926b] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. modify:started status incompatible"}],"requestID":"e5b62bdf-2c50-408e-8a14-04f11f1b926b"}
","requestId":"e5b62bdf-2c50-408e-8a14-04f11f1b926b","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=5f3432ec-0610-43a6-8fae-cd8c5b54ffe3] cannot determine cloud provider ","requestId":"5f3432ec-0610-43a6-8fae-cd8c5b54ffe3","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=5f3432ec-0610-43a6-8fae-cd8c5b54ffe3] cannot determine cloud provider ","requestId":"5f3432ec-0610-43a6-8fae-cd8c5b54ffe3","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_modify:failed_status[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8db5d7e8-1e45-49d2-8447-915c65219d83] cannot determine cloud provider ","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8db5d7e8-1e45-49d2-8447-915c65219d83] cannot determine cloud provider ","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=8db5d7e8-1e45-49d2-8447-915c65219d83] {"id":"45e8c516-3ae1-4d7f-bfcd-3ebda3335d47","timestamp":1706912519429,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:1937e660-a8e7-402a-bd87-0a55fe4b23f8","backupName":"test-backup-modify-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","tenantType":"PAAS"}}}","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: [0m[33m[0.267ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-modify-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=8db5d7e8-1e45-49d2-8447-915c65219d83] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=8db5d7e8-1e45-49d2-8447-915c65219d83] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. modify:failed status incompatible'.","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=8db5d7e8-1e45-49d2-8447-915c65219d83] {"id":"9f8841e9-e2e7-4827-bc3a-ac532371f32c","timestamp":1706912519440,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:1937e660-a8e7-402a-bd87-0a55fe4b23f8","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. modify:failed status incompatible\"}],\"requestID\":\"8db5d7e8-1e45-49d2-8447-915c65219d83\"}","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","tenantType":"PAAS"}}}","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=8db5d7e8-1e45-49d2-8447-915c65219d83] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. modify:failed status incompatible"}],"requestID":"8db5d7e8-1e45-49d2-8447-915c65219d83"}
","requestId":"8db5d7e8-1e45-49d2-8447-915c65219d83","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=2c2b897c-721d-43d2-aac0-0fb73bacb752] cannot determine cloud provider ","requestId":"2c2b897c-721d-43d2-aac0-0fb73bacb752","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=2c2b897c-721d-43d2-aac0-0fb73bacb752] cannot determine cloud provider ","requestId":"2c2b897c-721d-43d2-aac0-0fb73bacb752","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=2c2b897c-721d-43d2-aac0-0fb73bacb752] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"2c2b897c-721d-43d2-aac0-0fb73bacb752","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=2c2b897c-721d-43d2-aac0-0fb73bacb752] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. modify:warning status incompatible'.","requestId":"2c2b897c-721d-43d2-aac0-0fb73bacb752","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=2c2b897c-721d-43d2-aac0-0fb73bacb752] {"id":"0f746d73-f76f-4c48-9878-7e818c45f7db","timestamp":1706912519586,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:6e6ea91f-5fdb-41ed-8571-26a19a742993","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. modify:warning status incompatible\"}],\"requestID\":\"2c2b897c-721d-43d2-aac0-0fb73bacb752\"}","requestId":"2c2b897c-721d-43d2-aac0-0fb73bacb752","tenantType":"PAAS"}}}","requestId":"2c2b897c-721d-43d2-aac0-0fb73bacb752","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=2c2b897c-721d-43d2-aac0-0fb73bacb752] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. modify:warning status incompatible"}],"requestID":"2c2b897c-721d-43d2-aac0-0fb73bacb752"}
","requestId":"2c2b897c-721d-43d2-aac0-0fb73bacb752","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=6b667d9c-aacf-4b68-b829-9038e1645884] cannot determine cloud provider ","requestId":"6b667d9c-aacf-4b68-b829-9038e1645884","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=6b667d9c-aacf-4b68-b829-9038e1645884] cannot determine cloud provider ","requestId":"6b667d9c-aacf-4b68-b829-9038e1645884","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=6b667d9c-aacf-4b68-b829-9038e1645884] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"6b667d9c-aacf-4b68-b829-9038e1645884","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=6b667d9c-aacf-4b68-b829-9038e1645884] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. modify:succeeded status incompatible'.","requestId":"6b667d9c-aacf-4b68-b829-9038e1645884","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=6b667d9c-aacf-4b68-b829-9038e1645884] {"id":"0663ac91-edb2-4022-970b-202ab53597e8","timestamp":1706912519735,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:8a5efa7f-c1c4-44c0-ae42-e16adcb55903","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. modify:succeeded status incompatible\"}],\"requestID\":\"6b667d9c-aacf-4b68-b829-9038e1645884\"}","requestId":"6b667d9c-aacf-4b68-b829-9038e1645884","tenantType":"PAAS"}}}","requestId":"6b667d9c-aacf-4b68-b829-9038e1645884","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=6b667d9c-aacf-4b68-b829-9038e1645884] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. modify:succeeded status incompatible"}],"requestID":"6b667d9c-aacf-4b68-b829-9038e1645884"}
","requestId":"6b667d9c-aacf-4b68-b829-9038e1645884","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a] cannot determine cloud provider ","requestId":"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a] cannot determine cloud provider ","requestId":"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. unmount:started status incompatible'.","requestId":"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a] {"id":"e34441cd-21da-496a-bb53-05372167b286","timestamp":1706912519878,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:b2aa7397-1b3b-4571-855b-e6e9b65259be","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. unmount:started status incompatible\"}],\"requestID\":\"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a\"}","requestId":"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a","tenantType":"PAAS"}}}","requestId":"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. unmount:started status incompatible"}],"requestID":"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a"}
","requestId":"d5f89cb9-2c5a-44f3-ba1a-3d652d6d276a","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c302e263-d025-41b7-9849-e7fc8098bfed] cannot determine cloud provider ","requestId":"c302e263-d025-41b7-9849-e7fc8098bfed","timestamp":"2024-02-02T22:21:59Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c302e263-d025-41b7-9849-e7fc8098bfed] cannot determine cloud provider ","requestId":"c302e263-d025-41b7-9849-e7fc8098bfed","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=c302e263-d025-41b7-9849-e7fc8098bfed] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"c302e263-d025-41b7-9849-e7fc8098bfed","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=c302e263-d025-41b7-9849-e7fc8098bfed] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. unmount:finished status incompatible'.","requestId":"c302e263-d025-41b7-9849-e7fc8098bfed","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=c302e263-d025-41b7-9849-e7fc8098bfed] {"id":"2519cc8c-e21c-4bfa-9f98-75da32b06c07","timestamp":1706912520024,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:c5b41e4b-669e-4499-8ce4-5767011e6878","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. unmount:finished status incompatible\"}],\"requestID\":\"c302e263-d025-41b7-9849-e7fc8098bfed\"}","requestId":"c302e263-d025-41b7-9849-e7fc8098bfed","tenantType":"PAAS"}}}","requestId":"c302e263-d025-41b7-9849-e7fc8098bfed","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=c302e263-d025-41b7-9849-e7fc8098bfed] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. unmount:finished status incompatible"}],"requestID":"c302e263-d025-41b7-9849-e7fc8098bfed"}
","requestId":"c302e263-d025-41b7-9849-e7fc8098bfed","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_unmount:failed_status[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=3854c57c-b48b-4c14-8f91-575aad2575d3] cannot determine cloud provider ","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=3854c57c-b48b-4c14-8f91-575aad2575d3] cannot determine cloud provider ","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=3854c57c-b48b-4c14-8f91-575aad2575d3] {"id":"264d1faf-94a1-4a61-899e-0fa480c6cd7b","timestamp":1706912520154,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d60dd1d9-5dcf-42d7-bd40-e66f92971d14","backupName":"test-backup-unmount-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","tenantType":"PAAS"}}}","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: [0m[33m[0.239ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-unmount-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=3854c57c-b48b-4c14-8f91-575aad2575d3] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=3854c57c-b48b-4c14-8f91-575aad2575d3] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. unmount:failed status incompatible'.","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=3854c57c-b48b-4c14-8f91-575aad2575d3] {"id":"56af7e2c-cd0a-461a-b18b-f570b91e5c08","timestamp":1706912520161,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d60dd1d9-5dcf-42d7-bd40-e66f92971d14","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. unmount:failed status incompatible\"}],\"requestID\":\"3854c57c-b48b-4c14-8f91-575aad2575d3\"}","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","tenantType":"PAAS"}}}","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=3854c57c-b48b-4c14-8f91-575aad2575d3] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. unmount:failed status incompatible"}],"requestID":"3854c57c-b48b-4c14-8f91-575aad2575d3"}
","requestId":"3854c57c-b48b-4c14-8f91-575aad2575d3","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8d8b5243-2887-47a0-93fd-735971db375e] cannot determine cloud provider ","requestId":"8d8b5243-2887-47a0-93fd-735971db375e","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8d8b5243-2887-47a0-93fd-735971db375e] cannot determine cloud provider ","requestId":"8d8b5243-2887-47a0-93fd-735971db375e","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=8d8b5243-2887-47a0-93fd-735971db375e] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"8d8b5243-2887-47a0-93fd-735971db375e","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=8d8b5243-2887-47a0-93fd-735971db375e] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. unmount:warning status incompatible'.","requestId":"8d8b5243-2887-47a0-93fd-735971db375e","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=8d8b5243-2887-47a0-93fd-735971db375e] {"id":"4ad26242-870c-44be-bae0-925d806688f7","timestamp":1706912520303,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:de2db308-2be4-4050-a3d2-c24a517629cb","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. unmount:warning status incompatible\"}],\"requestID\":\"8d8b5243-2887-47a0-93fd-735971db375e\"}","requestId":"8d8b5243-2887-47a0-93fd-735971db375e","tenantType":"PAAS"}}}","requestId":"8d8b5243-2887-47a0-93fd-735971db375e","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=8d8b5243-2887-47a0-93fd-735971db375e] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. unmount:warning status incompatible"}],"requestID":"8d8b5243-2887-47a0-93fd-735971db375e"}
","requestId":"8d8b5243-2887-47a0-93fd-735971db375e","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=3b2940ba-cd8b-434c-8a49-66405075bc09] cannot determine cloud provider ","requestId":"3b2940ba-cd8b-434c-8a49-66405075bc09","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=3b2940ba-cd8b-434c-8a49-66405075bc09] cannot determine cloud provider ","requestId":"3b2940ba-cd8b-434c-8a49-66405075bc09","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=3b2940ba-cd8b-434c-8a49-66405075bc09] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"3b2940ba-cd8b-434c-8a49-66405075bc09","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=3b2940ba-cd8b-434c-8a49-66405075bc09] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. unmount:succeeded status incompatible'.","requestId":"3b2940ba-cd8b-434c-8a49-66405075bc09","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=3b2940ba-cd8b-434c-8a49-66405075bc09] {"id":"ec0efee5-70f3-4bad-b95a-e3d5850a0d66","timestamp":1706912520444,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d93f5449-e184-4236-9bdd-b1bece2f73dc","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. unmount:succeeded status incompatible\"}],\"requestID\":\"3b2940ba-cd8b-434c-8a49-66405075bc09\"}","requestId":"3b2940ba-cd8b-434c-8a49-66405075bc09","tenantType":"PAAS"}}}","requestId":"3b2940ba-cd8b-434c-8a49-66405075bc09","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=3b2940ba-cd8b-434c-8a49-66405075bc09] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. unmount:succeeded status incompatible"}],"requestID":"3b2940ba-cd8b-434c-8a49-66405075bc09"}
","requestId":"3b2940ba-cd8b-434c-8a49-66405075bc09","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=1539e502-36af-4bfd-8c03-e47b8560a5e7] cannot determine cloud provider ","requestId":"1539e502-36af-4bfd-8c03-e47b8560a5e7","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=1539e502-36af-4bfd-8c03-e47b8560a5e7] cannot determine cloud provider ","requestId":"1539e502-36af-4bfd-8c03-e47b8560a5e7","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=1539e502-36af-4bfd-8c03-e47b8560a5e7] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"1539e502-36af-4bfd-8c03-e47b8560a5e7","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=1539e502-36af-4bfd-8c03-e47b8560a5e7] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provisionRequestValidation:started status incompatible'.","requestId":"1539e502-36af-4bfd-8c03-e47b8560a5e7","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=1539e502-36af-4bfd-8c03-e47b8560a5e7] {"id":"612007fc-abc9-4e31-a827-eb62cdf3cfd3","timestamp":1706912520589,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:b4992243-acd3-43b9-8694-633b3ed0aa68","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provisionRequestValidation:started status incompatible\"}],\"requestID\":\"1539e502-36af-4bfd-8c03-e47b8560a5e7\"}","requestId":"1539e502-36af-4bfd-8c03-e47b8560a5e7","tenantType":"PAAS"}}}","requestId":"1539e502-36af-4bfd-8c03-e47b8560a5e7","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=1539e502-36af-4bfd-8c03-e47b8560a5e7] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provisionRequestValidation:started status incompatible"}],"requestID":"1539e502-36af-4bfd-8c03-e47b8560a5e7"}
","requestId":"1539e502-36af-4bfd-8c03-e47b8560a5e7","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=ddf4bb35-f4db-4dea-9e09-6446452309b5] cannot determine cloud provider ","requestId":"ddf4bb35-f4db-4dea-9e09-6446452309b5","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=ddf4bb35-f4db-4dea-9e09-6446452309b5] cannot determine cloud provider ","requestId":"ddf4bb35-f4db-4dea-9e09-6446452309b5","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=ddf4bb35-f4db-4dea-9e09-6446452309b5] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"ddf4bb35-f4db-4dea-9e09-6446452309b5","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=ddf4bb35-f4db-4dea-9e09-6446452309b5] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provisionRequestValidation:finished status incompatible'.","requestId":"ddf4bb35-f4db-4dea-9e09-6446452309b5","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=ddf4bb35-f4db-4dea-9e09-6446452309b5] {"id":"8bb94a73-8c10-4dfa-9b43-b8a1b3e0862c","timestamp":1706912520736,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:261fb032-d686-4619-ba7b-e078b9274fa8","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provisionRequestValidation:finished status incompatible\"}],\"requestID\":\"ddf4bb35-f4db-4dea-9e09-6446452309b5\"}","requestId":"ddf4bb35-f4db-4dea-9e09-6446452309b5","tenantType":"PAAS"}}}","requestId":"ddf4bb35-f4db-4dea-9e09-6446452309b5","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=ddf4bb35-f4db-4dea-9e09-6446452309b5] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provisionRequestValidation:finished status incompatible"}],"requestID":"ddf4bb35-f4db-4dea-9e09-6446452309b5"}
","requestId":"ddf4bb35-f4db-4dea-9e09-6446452309b5","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_provisionRequestValidation:failed_status[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=91b1241b-6b04-4e31-bc69-48c35f3fa870] cannot determine cloud provider ","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=91b1241b-6b04-4e31-bc69-48c35f3fa870] cannot determine cloud provider ","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=91b1241b-6b04-4e31-bc69-48c35f3fa870] {"id":"56d51daf-0542-434f-abee-65df450beea4","timestamp":1706912520868,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:65ed31ae-261f-4ae0-a4bf-f47a85d7b8b0","backupName":"test-backup-provisionRequestValidation-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","tenantType":"PAAS"}}}","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: [0m[33m[0.239ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-provisionRequestValidation-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=91b1241b-6b04-4e31-bc69-48c35f3fa870] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=91b1241b-6b04-4e31-bc69-48c35f3fa870] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provisionRequestValidation:failed status incompatible'.","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=91b1241b-6b04-4e31-bc69-48c35f3fa870] {"id":"7b38a211-5205-4abd-8cd8-79f263ea5a07","timestamp":1706912520876,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:65ed31ae-261f-4ae0-a4bf-f47a85d7b8b0","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provisionRequestValidation:failed status incompatible\"}],\"requestID\":\"91b1241b-6b04-4e31-bc69-48c35f3fa870\"}","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","tenantType":"PAAS"}}}","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=91b1241b-6b04-4e31-bc69-48c35f3fa870] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provisionRequestValidation:failed status incompatible"}],"requestID":"91b1241b-6b04-4e31-bc69-48c35f3fa870"}
","requestId":"91b1241b-6b04-4e31-bc69-48c35f3fa870","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=461dc272-1fd7-4e7b-b720-5868409de1bf] cannot determine cloud provider ","requestId":"461dc272-1fd7-4e7b-b720-5868409de1bf","timestamp":"2024-02-02T22:22:00Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=461dc272-1fd7-4e7b-b720-5868409de1bf] cannot determine cloud provider ","requestId":"461dc272-1fd7-4e7b-b720-5868409de1bf","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=461dc272-1fd7-4e7b-b720-5868409de1bf] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"461dc272-1fd7-4e7b-b720-5868409de1bf","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=461dc272-1fd7-4e7b-b720-5868409de1bf] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provisionRequestValidation:warning status incompatible'.","requestId":"461dc272-1fd7-4e7b-b720-5868409de1bf","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=461dc272-1fd7-4e7b-b720-5868409de1bf] {"id":"881fe914-848b-4bad-ae7f-52fbabed7cb3","timestamp":1706912521166,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:b03682f3-e735-44f3-b4a3-37595399d5e0","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provisionRequestValidation:warning status incompatible\"}],\"requestID\":\"461dc272-1fd7-4e7b-b720-5868409de1bf\"}","requestId":"461dc272-1fd7-4e7b-b720-5868409de1bf","tenantType":"PAAS"}}}","requestId":"461dc272-1fd7-4e7b-b720-5868409de1bf","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=461dc272-1fd7-4e7b-b720-5868409de1bf] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provisionRequestValidation:warning status incompatible"}],"requestID":"461dc272-1fd7-4e7b-b720-5868409de1bf"}
","requestId":"461dc272-1fd7-4e7b-b720-5868409de1bf","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=541037f8-2c91-447e-a5ad-3c4c46bfaf6c] cannot determine cloud provider ","requestId":"541037f8-2c91-447e-a5ad-3c4c46bfaf6c","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=541037f8-2c91-447e-a5ad-3c4c46bfaf6c] cannot determine cloud provider ","requestId":"541037f8-2c91-447e-a5ad-3c4c46bfaf6c","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=541037f8-2c91-447e-a5ad-3c4c46bfaf6c] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"541037f8-2c91-447e-a5ad-3c4c46bfaf6c","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=541037f8-2c91-447e-a5ad-3c4c46bfaf6c] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provisionRequestValidation:succeeded status incompatible'.","requestId":"541037f8-2c91-447e-a5ad-3c4c46bfaf6c","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=541037f8-2c91-447e-a5ad-3c4c46bfaf6c] {"id":"9bd099f5-3207-44b3-8f31-abf4e8d3f9ef","timestamp":1706912521590,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:2ffd982f-47e9-4349-86c3-92c5dccf390a","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provisionRequestValidation:succeeded status incompatible\"}],\"requestID\":\"541037f8-2c91-447e-a5ad-3c4c46bfaf6c\"}","requestId":"541037f8-2c91-447e-a5ad-3c4c46bfaf6c","tenantType":"PAAS"}}}","requestId":"541037f8-2c91-447e-a5ad-3c4c46bfaf6c","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=541037f8-2c91-447e-a5ad-3c4c46bfaf6c] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provisionRequestValidation:succeeded status incompatible"}],"requestID":"541037f8-2c91-447e-a5ad-3c4c46bfaf6c"}
","requestId":"541037f8-2c91-447e-a5ad-3c4c46bfaf6c","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=da50176d-131a-49d4-82ec-bcd74d29a5ab] cannot determine cloud provider ","requestId":"da50176d-131a-49d4-82ec-bcd74d29a5ab","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=da50176d-131a-49d4-82ec-bcd74d29a5ab] cannot determine cloud provider ","requestId":"da50176d-131a-49d4-82ec-bcd74d29a5ab","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=da50176d-131a-49d4-82ec-bcd74d29a5ab] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"da50176d-131a-49d4-82ec-bcd74d29a5ab","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=da50176d-131a-49d4-82ec-bcd74d29a5ab] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. preInstallValidation:started status incompatible'.","requestId":"da50176d-131a-49d4-82ec-bcd74d29a5ab","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=da50176d-131a-49d4-82ec-bcd74d29a5ab] {"id":"293ff116-354b-4d9f-bb24-7d322b8b0a66","timestamp":1706912521943,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:a037869d-0339-4e05-a0d0-88bb99a54383","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. preInstallValidation:started status incompatible\"}],\"requestID\":\"da50176d-131a-49d4-82ec-bcd74d29a5ab\"}","requestId":"da50176d-131a-49d4-82ec-bcd74d29a5ab","tenantType":"PAAS"}}}","requestId":"da50176d-131a-49d4-82ec-bcd74d29a5ab","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=da50176d-131a-49d4-82ec-bcd74d29a5ab] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. preInstallValidation:started status incompatible"}],"requestID":"da50176d-131a-49d4-82ec-bcd74d29a5ab"}
","requestId":"da50176d-131a-49d4-82ec-bcd74d29a5ab","timestamp":"2024-02-02T22:22:01Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a531f450-a197-414e-8846-4bdb13884541] cannot determine cloud provider ","requestId":"a531f450-a197-414e-8846-4bdb13884541","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a531f450-a197-414e-8846-4bdb13884541] cannot determine cloud provider ","requestId":"a531f450-a197-414e-8846-4bdb13884541","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a531f450-a197-414e-8846-4bdb13884541] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a531f450-a197-414e-8846-4bdb13884541","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=a531f450-a197-414e-8846-4bdb13884541] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. preInstallValidation:finished status incompatible'.","requestId":"a531f450-a197-414e-8846-4bdb13884541","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a531f450-a197-414e-8846-4bdb13884541] {"id":"88a8f8d4-e98d-4243-b860-37b4c9a02b3d","timestamp":1706912522141,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:04eae908-ed2e-4190-972f-468db5037153","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. preInstallValidation:finished status incompatible\"}],\"requestID\":\"a531f450-a197-414e-8846-4bdb13884541\"}","requestId":"a531f450-a197-414e-8846-4bdb13884541","tenantType":"PAAS"}}}","requestId":"a531f450-a197-414e-8846-4bdb13884541","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=a531f450-a197-414e-8846-4bdb13884541] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. preInstallValidation:finished status incompatible"}],"requestID":"a531f450-a197-414e-8846-4bdb13884541"}
","requestId":"a531f450-a197-414e-8846-4bdb13884541","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_preInstallValidation:failed_status[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=55846655-9d9d-4f28-b79f-2a8712c3edb0] cannot determine cloud provider ","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=55846655-9d9d-4f28-b79f-2a8712c3edb0] cannot determine cloud provider ","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=55846655-9d9d-4f28-b79f-2a8712c3edb0] {"id":"5477e769-8135-4574-8123-91218141ba5b","timestamp":1706912522273,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:e53d1f9d-4395-4401-8f84-d256b6199e6d","backupName":"test-backup-preInstallValidation-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","tenantType":"PAAS"}}}","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=55846655-9d9d-4f28-b79f-2a8712c3edb0] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=55846655-9d9d-4f28-b79f-2a8712c3edb0] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. preInstallValidation:failed status incompatible'.","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: [0m[33m[0.251ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-preInstallValidation-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=55846655-9d9d-4f28-b79f-2a8712c3edb0] {"id":"92e76ba4-9417-4edb-acda-08dfd7f4b1fb","timestamp":1706912522288,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:e53d1f9d-4395-4401-8f84-d256b6199e6d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. preInstallValidation:failed status incompatible\"}],\"requestID\":\"55846655-9d9d-4f28-b79f-2a8712c3edb0\"}","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","tenantType":"PAAS"}}}","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=55846655-9d9d-4f28-b79f-2a8712c3edb0] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. preInstallValidation:failed status incompatible"}],"requestID":"55846655-9d9d-4f28-b79f-2a8712c3edb0"}
","requestId":"55846655-9d9d-4f28-b79f-2a8712c3edb0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=ceb18486-5db0-4dce-b065-4aafee6e044b] cannot determine cloud provider ","requestId":"ceb18486-5db0-4dce-b065-4aafee6e044b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=ceb18486-5db0-4dce-b065-4aafee6e044b] cannot determine cloud provider ","requestId":"ceb18486-5db0-4dce-b065-4aafee6e044b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=ceb18486-5db0-4dce-b065-4aafee6e044b] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"ceb18486-5db0-4dce-b065-4aafee6e044b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=ceb18486-5db0-4dce-b065-4aafee6e044b] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. preInstallValidation:warning status incompatible'.","requestId":"ceb18486-5db0-4dce-b065-4aafee6e044b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=ceb18486-5db0-4dce-b065-4aafee6e044b] {"id":"7bd81076-8294-4476-b1c1-e6d6d25b5c04","timestamp":1706912522433,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3ae5447e-91e7-48b6-bacd-da4bcdb5d9c5","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. preInstallValidation:warning status incompatible\"}],\"requestID\":\"ceb18486-5db0-4dce-b065-4aafee6e044b\"}","requestId":"ceb18486-5db0-4dce-b065-4aafee6e044b","tenantType":"PAAS"}}}","requestId":"ceb18486-5db0-4dce-b065-4aafee6e044b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=ceb18486-5db0-4dce-b065-4aafee6e044b] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. preInstallValidation:warning status incompatible"}],"requestID":"ceb18486-5db0-4dce-b065-4aafee6e044b"}
","requestId":"ceb18486-5db0-4dce-b065-4aafee6e044b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=1918e957-40e0-4551-a741-959724cf60a1] cannot determine cloud provider ","requestId":"1918e957-40e0-4551-a741-959724cf60a1","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=1918e957-40e0-4551-a741-959724cf60a1] cannot determine cloud provider ","requestId":"1918e957-40e0-4551-a741-959724cf60a1","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=1918e957-40e0-4551-a741-959724cf60a1] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"1918e957-40e0-4551-a741-959724cf60a1","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=1918e957-40e0-4551-a741-959724cf60a1] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. preInstallValidation:succeeded status incompatible'.","requestId":"1918e957-40e0-4551-a741-959724cf60a1","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=1918e957-40e0-4551-a741-959724cf60a1] {"id":"69729c4e-e62a-42a9-8a2f-9a17e8cb305e","timestamp":1706912522575,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:c914eced-88e0-4b43-8aa4-287c69bb5c6d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. preInstallValidation:succeeded status incompatible\"}],\"requestID\":\"1918e957-40e0-4551-a741-959724cf60a1\"}","requestId":"1918e957-40e0-4551-a741-959724cf60a1","tenantType":"PAAS"}}}","requestId":"1918e957-40e0-4551-a741-959724cf60a1","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=1918e957-40e0-4551-a741-959724cf60a1] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. preInstallValidation:succeeded status incompatible"}],"requestID":"1918e957-40e0-4551-a741-959724cf60a1"}
","requestId":"1918e957-40e0-4551-a741-959724cf60a1","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=70a95502-93e0-4ccf-8cf7-d5e9aa5f013b] cannot determine cloud provider ","requestId":"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=70a95502-93e0-4ccf-8cf7-d5e9aa5f013b] cannot determine cloud provider ","requestId":"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=70a95502-93e0-4ccf-8cf7-d5e9aa5f013b] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=70a95502-93e0-4ccf-8cf7-d5e9aa5f013b] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. shutDownForBackup:started status incompatible'.","requestId":"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=70a95502-93e0-4ccf-8cf7-d5e9aa5f013b] {"id":"c87fb14f-b07f-4307-b143-f53ead1765e9","timestamp":1706912522720,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:bfefbff5-7d66-41cb-87e8-4984b214c637","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. shutDownForBackup:started status incompatible\"}],\"requestID\":\"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b\"}","requestId":"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b","tenantType":"PAAS"}}}","requestId":"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=70a95502-93e0-4ccf-8cf7-d5e9aa5f013b] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. shutDownForBackup:started status incompatible"}],"requestID":"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b"}
","requestId":"70a95502-93e0-4ccf-8cf7-d5e9aa5f013b","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cd256c8c-3775-45f1-b92d-d3c45fdb8da0] cannot determine cloud provider ","requestId":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cd256c8c-3775-45f1-b92d-d3c45fdb8da0] cannot determine cloud provider ","requestId":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cd256c8c-3775-45f1-b92d-d3c45fdb8da0] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=cd256c8c-3775-45f1-b92d-d3c45fdb8da0] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. shutDownForBackup:finished status incompatible'.","requestId":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cd256c8c-3775-45f1-b92d-d3c45fdb8da0] {"id":"ef1b9b73-d455-4523-a6da-1774564cfc23","timestamp":1706912522864,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d069365b-4c07-4848-8a37-402cb06a46e7","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. shutDownForBackup:finished status incompatible\"}],\"requestID\":\"cd256c8c-3775-45f1-b92d-d3c45fdb8da0\"}","requestId":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0","tenantType":"PAAS"}}}","requestId":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=cd256c8c-3775-45f1-b92d-d3c45fdb8da0] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. shutDownForBackup:finished status incompatible"}],"requestID":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0"}
","requestId":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_shutDownForBackup:failed_status[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=fdc9bb02-5ae5-4730-a711-8518e3541140] cannot determine cloud provider ","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=fdc9bb02-5ae5-4730-a711-8518e3541140] cannot determine cloud provider ","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=fdc9bb02-5ae5-4730-a711-8518e3541140] {"id":"d3683e09-bf77-4c4e-9183-955e6420294e","timestamp":1706912522998,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:147c9f5b-66f1-4988-bcf5-36f7762ba9b1","backupName":"test-backup-shutDownForBackup-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","tenantType":"PAAS"}}}","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","timestamp":"2024-02-02T22:22:02Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: [0m[33m[0.221ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-shutDownForBackup-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=fdc9bb02-5ae5-4730-a711-8518e3541140] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=fdc9bb02-5ae5-4730-a711-8518e3541140] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. shutDownForBackup:failed status incompatible'.","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=fdc9bb02-5ae5-4730-a711-8518e3541140] {"id":"b3ede088-3810-4a08-be7d-8deeb03d6a93","timestamp":1706912523007,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:147c9f5b-66f1-4988-bcf5-36f7762ba9b1","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. shutDownForBackup:failed status incompatible\"}],\"requestID\":\"fdc9bb02-5ae5-4730-a711-8518e3541140\"}","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","tenantType":"PAAS"}}}","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=fdc9bb02-5ae5-4730-a711-8518e3541140] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. shutDownForBackup:failed status incompatible"}],"requestID":"fdc9bb02-5ae5-4730-a711-8518e3541140"}
","requestId":"fdc9bb02-5ae5-4730-a711-8518e3541140","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c27bd676-142c-45f5-ae82-62850a6398cb] cannot determine cloud provider ","requestId":"c27bd676-142c-45f5-ae82-62850a6398cb","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c27bd676-142c-45f5-ae82-62850a6398cb] cannot determine cloud provider ","requestId":"c27bd676-142c-45f5-ae82-62850a6398cb","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=c27bd676-142c-45f5-ae82-62850a6398cb] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"c27bd676-142c-45f5-ae82-62850a6398cb","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=c27bd676-142c-45f5-ae82-62850a6398cb] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. shutDownForBackup:warning status incompatible'.","requestId":"c27bd676-142c-45f5-ae82-62850a6398cb","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=c27bd676-142c-45f5-ae82-62850a6398cb] {"id":"1e78bff1-e389-43a0-8ae2-d4763b9dc4c6","timestamp":1706912523170,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3c7ad9a1-1d34-4334-8412-972a78a66528","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. shutDownForBackup:warning status incompatible\"}],\"requestID\":\"c27bd676-142c-45f5-ae82-62850a6398cb\"}","requestId":"c27bd676-142c-45f5-ae82-62850a6398cb","tenantType":"PAAS"}}}","requestId":"c27bd676-142c-45f5-ae82-62850a6398cb","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=c27bd676-142c-45f5-ae82-62850a6398cb] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. shutDownForBackup:warning status incompatible"}],"requestID":"c27bd676-142c-45f5-ae82-62850a6398cb"}
","requestId":"c27bd676-142c-45f5-ae82-62850a6398cb","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=24b794d9-18aa-48be-889f-bb72bc5b7f2e] cannot determine cloud provider ","requestId":"24b794d9-18aa-48be-889f-bb72bc5b7f2e","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=24b794d9-18aa-48be-889f-bb72bc5b7f2e] cannot determine cloud provider ","requestId":"24b794d9-18aa-48be-889f-bb72bc5b7f2e","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=24b794d9-18aa-48be-889f-bb72bc5b7f2e] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"24b794d9-18aa-48be-889f-bb72bc5b7f2e","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=24b794d9-18aa-48be-889f-bb72bc5b7f2e] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. shutDownForBackup:succeeded status incompatible'.","requestId":"24b794d9-18aa-48be-889f-bb72bc5b7f2e","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=24b794d9-18aa-48be-889f-bb72bc5b7f2e] {"id":"63892528-ffd6-4657-bf88-3ea59659a88c","timestamp":1706912523321,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:1222bfe8-5267-4c96-9b74-4692566d1303","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. shutDownForBackup:succeeded status incompatible\"}],\"requestID\":\"24b794d9-18aa-48be-889f-bb72bc5b7f2e\"}","requestId":"24b794d9-18aa-48be-889f-bb72bc5b7f2e","tenantType":"PAAS"}}}","requestId":"24b794d9-18aa-48be-889f-bb72bc5b7f2e","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=24b794d9-18aa-48be-889f-bb72bc5b7f2e] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. shutDownForBackup:succeeded status incompatible"}],"requestID":"24b794d9-18aa-48be-889f-bb72bc5b7f2e"}
","requestId":"24b794d9-18aa-48be-889f-bb72bc5b7f2e","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764] cannot determine cloud provider ","requestId":"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764] cannot determine cloud provider ","requestId":"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restartAfterBackup:started status incompatible'.","requestId":"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764] {"id":"191cb09e-9182-4f38-86d0-74d062c41cbd","timestamp":1706912523478,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3b0558ff-9cc1-4596-9c39-dfca6332590f","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restartAfterBackup:started status incompatible\"}],\"requestID\":\"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764\"}","requestId":"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764","tenantType":"PAAS"}}}","requestId":"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restartAfterBackup:started status incompatible"}],"requestID":"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764"}
","requestId":"f55ffc16-06b2-42e4-8ce5-f0dc9f8c6764","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5] cannot determine cloud provider ","requestId":"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5] cannot determine cloud provider ","requestId":"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restartAfterBackup:finished status incompatible'.","requestId":"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5] {"id":"2220fe90-806c-436e-b370-cdf7fd7f6af8","timestamp":1706912523621,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:be7116ce-fa4a-4fe2-b0cf-c2fed4dc9b1d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restartAfterBackup:finished status incompatible\"}],\"requestID\":\"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5\"}","requestId":"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5","tenantType":"PAAS"}}}","requestId":"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restartAfterBackup:finished status incompatible"}],"requestID":"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5"}
","requestId":"82434b89-1ed5-4dd3-adfd-fb38a6f0b8a5","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_restartAfterBackup:failed_status[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1] cannot determine cloud provider ","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1] cannot determine cloud provider ","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1] {"id":"3999b3f7-cc51-4f3f-9d61-92fb5ec8d32c","timestamp":1706912523765,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:92edba32-5fa0-4514-a737-d5b897fd95a4","backupName":"test-backup-restartAfterBackup-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","tenantType":"PAAS"}}}","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: [0m[33m[0.289ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-restartAfterBackup-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restartAfterBackup:failed status incompatible'.","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1] {"id":"3664e79f-5fb8-4c94-9967-32c8e19387d8","timestamp":1706912523783,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:92edba32-5fa0-4514-a737-d5b897fd95a4","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restartAfterBackup:failed status incompatible\"}],\"requestID\":\"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1\"}","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","tenantType":"PAAS"}}}","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restartAfterBackup:failed status incompatible"}],"requestID":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1"}
","requestId":"bb75c4e0-d7a0-477c-a17e-bcc35c1ecbf1","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2] cannot determine cloud provider ","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2] cannot determine cloud provider ","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restartAfterBackup:warning status incompatible'.","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2] {"id":"6b229e21-eb78-446c-b88f-ffb8a32c03ec","timestamp":1706912523936,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:11f968b3-ae43-423c-8651-ae105722d680","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restartAfterBackup:warning status incompatible\"}],\"requestID\":\"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2\"}","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","tenantType":"PAAS"}}}","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restartAfterBackup:warning status incompatible"}],"requestID":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2"}
","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","timestamp":"2024-02-02T22:22:03Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cc1a1d48-46ac-46d4-99b8-779f8991cc20] cannot determine cloud provider ","requestId":"cc1a1d48-46ac-46d4-99b8-779f8991cc20","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cc1a1d48-46ac-46d4-99b8-779f8991cc20] cannot determine cloud provider ","requestId":"cc1a1d48-46ac-46d4-99b8-779f8991cc20","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cc1a1d48-46ac-46d4-99b8-779f8991cc20] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cc1a1d48-46ac-46d4-99b8-779f8991cc20","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=cc1a1d48-46ac-46d4-99b8-779f8991cc20] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restartAfterBackup:succeeded status incompatible'.","requestId":"cc1a1d48-46ac-46d4-99b8-779f8991cc20","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cc1a1d48-46ac-46d4-99b8-779f8991cc20] {"id":"131fb9c6-e4a2-48fc-870f-20dd3557caad","timestamp":1706912524084,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:0ee71860-72c7-4b54-aed4-1ed0b2e01fdb","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restartAfterBackup:succeeded status incompatible\"}],\"requestID\":\"cc1a1d48-46ac-46d4-99b8-779f8991cc20\"}","requestId":"cc1a1d48-46ac-46d4-99b8-779f8991cc20","tenantType":"PAAS"}}}","requestId":"cc1a1d48-46ac-46d4-99b8-779f8991cc20","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=cc1a1d48-46ac-46d4-99b8-779f8991cc20] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restartAfterBackup:succeeded status incompatible"}],"requestID":"cc1a1d48-46ac-46d4-99b8-779f8991cc20"}
","requestId":"cc1a1d48-46ac-46d4-99b8-779f8991cc20","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=3cbf8403-331c-40b6-9da1-397b9eab01ca] cannot determine cloud provider ","requestId":"3cbf8403-331c-40b6-9da1-397b9eab01ca","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=3cbf8403-331c-40b6-9da1-397b9eab01ca] cannot determine cloud provider ","requestId":"3cbf8403-331c-40b6-9da1-397b9eab01ca","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=3cbf8403-331c-40b6-9da1-397b9eab01ca] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"3cbf8403-331c-40b6-9da1-397b9eab01ca","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=3cbf8403-331c-40b6-9da1-397b9eab01ca] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. backup:started status incompatible'.","requestId":"3cbf8403-331c-40b6-9da1-397b9eab01ca","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=3cbf8403-331c-40b6-9da1-397b9eab01ca] {"id":"3bc6cb61-3ee2-4238-baa4-9ae4cba525d8","timestamp":1706912524231,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:a6e2481e-3d1c-4d1d-9307-c7914f716801","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. backup:started status incompatible\"}],\"requestID\":\"3cbf8403-331c-40b6-9da1-397b9eab01ca\"}","requestId":"3cbf8403-331c-40b6-9da1-397b9eab01ca","tenantType":"PAAS"}}}","requestId":"3cbf8403-331c-40b6-9da1-397b9eab01ca","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=3cbf8403-331c-40b6-9da1-397b9eab01ca] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. backup:started status incompatible"}],"requestID":"3cbf8403-331c-40b6-9da1-397b9eab01ca"}
","requestId":"3cbf8403-331c-40b6-9da1-397b9eab01ca","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=b5c50dcb-c578-4088-895c-18de3770ba97] cannot determine cloud provider ","requestId":"b5c50dcb-c578-4088-895c-18de3770ba97","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=b5c50dcb-c578-4088-895c-18de3770ba97] cannot determine cloud provider ","requestId":"b5c50dcb-c578-4088-895c-18de3770ba97","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=b5c50dcb-c578-4088-895c-18de3770ba97] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"b5c50dcb-c578-4088-895c-18de3770ba97","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=b5c50dcb-c578-4088-895c-18de3770ba97] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. backup:finished status incompatible'.","requestId":"b5c50dcb-c578-4088-895c-18de3770ba97","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=b5c50dcb-c578-4088-895c-18de3770ba97] {"id":"75cb274b-9a67-4014-af5c-a117db9621ac","timestamp":1706912524375,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:fa9dbc5d-a4a6-49e2-b616-c115def24315","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. backup:finished status incompatible\"}],\"requestID\":\"b5c50dcb-c578-4088-895c-18de3770ba97\"}","requestId":"b5c50dcb-c578-4088-895c-18de3770ba97","tenantType":"PAAS"}}}","requestId":"b5c50dcb-c578-4088-895c-18de3770ba97","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=b5c50dcb-c578-4088-895c-18de3770ba97] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. backup:finished status incompatible"}],"requestID":"b5c50dcb-c578-4088-895c-18de3770ba97"}
","requestId":"b5c50dcb-c578-4088-895c-18de3770ba97","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_backup:failed_status[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3] cannot determine cloud provider ","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3] cannot determine cloud provider ","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3] {"id":"8b7976bc-249b-4b63-ac10-7cde0bf4fead","timestamp":1706912524510,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:06992c60-790e-4151-916b-ff436d480a9d","backupName":"test-backup-backup-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","tenantType":"PAAS"}}}","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. backup:failed status incompatible'.","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: [0m[33m[0.355ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-backup-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3] {"id":"b7203c9c-538d-43e3-9e62-410410a1d5a1","timestamp":1706912524709,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:06992c60-790e-4151-916b-ff436d480a9d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. backup:failed status incompatible\"}],\"requestID\":\"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3\"}","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","tenantType":"PAAS"}}}","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. backup:failed status incompatible"}],"requestID":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3"}
","requestId":"9cb82ea0-f9e3-4479-a49f-d4ef3744d7d3","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=2b27da7d-07d5-4026-b757-4ddb72153c5a] cannot determine cloud provider ","requestId":"2b27da7d-07d5-4026-b757-4ddb72153c5a","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=2b27da7d-07d5-4026-b757-4ddb72153c5a] cannot determine cloud provider ","requestId":"2b27da7d-07d5-4026-b757-4ddb72153c5a","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=2b27da7d-07d5-4026-b757-4ddb72153c5a] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"2b27da7d-07d5-4026-b757-4ddb72153c5a","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=2b27da7d-07d5-4026-b757-4ddb72153c5a] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. backup:warning status incompatible'.","requestId":"2b27da7d-07d5-4026-b757-4ddb72153c5a","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=2b27da7d-07d5-4026-b757-4ddb72153c5a] {"id":"10fa9a2a-be63-480c-804d-04ab5456707f","timestamp":1706912524861,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:86e3dd69-e796-48f1-ae22-bb3e0de518e0","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. backup:warning status incompatible\"}],\"requestID\":\"2b27da7d-07d5-4026-b757-4ddb72153c5a\"}","requestId":"2b27da7d-07d5-4026-b757-4ddb72153c5a","tenantType":"PAAS"}}}","requestId":"2b27da7d-07d5-4026-b757-4ddb72153c5a","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=2b27da7d-07d5-4026-b757-4ddb72153c5a] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. backup:warning status incompatible"}],"requestID":"2b27da7d-07d5-4026-b757-4ddb72153c5a"}
","requestId":"2b27da7d-07d5-4026-b757-4ddb72153c5a","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6] cannot determine cloud provider ","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6] cannot determine cloud provider ","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","timestamp":"2024-02-02T22:22:04Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. backup:succeeded status incompatible'.","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6] {"id":"c331f850-d8ca-4725-8559-860044cef4bb","timestamp":1706912525016,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3b1ce31a-5c17-45c6-82dc-4499789b9a33","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. backup:succeeded status incompatible\"}],\"requestID\":\"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6\"}","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","tenantType":"PAAS"}}}","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. backup:succeeded status incompatible"}],"requestID":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6"}
","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8190ca19-8172-4f42-8194-af600a6add6f] cannot determine cloud provider ","requestId":"8190ca19-8172-4f42-8194-af600a6add6f","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8190ca19-8172-4f42-8194-af600a6add6f] cannot determine cloud provider ","requestId":"8190ca19-8172-4f42-8194-af600a6add6f","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=8190ca19-8172-4f42-8194-af600a6add6f] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"8190ca19-8172-4f42-8194-af600a6add6f","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=8190ca19-8172-4f42-8194-af600a6add6f] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restore:started status incompatible'.","requestId":"8190ca19-8172-4f42-8194-af600a6add6f","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=8190ca19-8172-4f42-8194-af600a6add6f] {"id":"0d7ec166-11aa-41fd-86a9-dc4bd83d54aa","timestamp":1706912525160,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:15df05f8-0dab-46a2-b292-65bbf4e67b9d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restore:started status incompatible\"}],\"requestID\":\"8190ca19-8172-4f42-8194-af600a6add6f\"}","requestId":"8190ca19-8172-4f42-8194-af600a6add6f","tenantType":"PAAS"}}}","requestId":"8190ca19-8172-4f42-8194-af600a6add6f","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=8190ca19-8172-4f42-8194-af600a6add6f] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restore:started status incompatible"}],"requestID":"8190ca19-8172-4f42-8194-af600a6add6f"}
","requestId":"8190ca19-8172-4f42-8194-af600a6add6f","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a78f7426-d7b8-476b-b86f-504dbf771982] cannot determine cloud provider ","requestId":"a78f7426-d7b8-476b-b86f-504dbf771982","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a78f7426-d7b8-476b-b86f-504dbf771982] cannot determine cloud provider ","requestId":"a78f7426-d7b8-476b-b86f-504dbf771982","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a78f7426-d7b8-476b-b86f-504dbf771982] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a78f7426-d7b8-476b-b86f-504dbf771982","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=a78f7426-d7b8-476b-b86f-504dbf771982] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restore:finished status incompatible'.","requestId":"a78f7426-d7b8-476b-b86f-504dbf771982","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a78f7426-d7b8-476b-b86f-504dbf771982] {"id":"85808b45-1f62-4a4a-b5a4-169d19af666e","timestamp":1706912525311,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3a19830c-3046-4a3d-883d-e0174f6ae77e","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restore:finished status incompatible\"}],\"requestID\":\"a78f7426-d7b8-476b-b86f-504dbf771982\"}","requestId":"a78f7426-d7b8-476b-b86f-504dbf771982","tenantType":"PAAS"}}}","requestId":"a78f7426-d7b8-476b-b86f-504dbf771982","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=a78f7426-d7b8-476b-b86f-504dbf771982] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restore:finished status incompatible"}],"requestID":"a78f7426-d7b8-476b-b86f-504dbf771982"}
","requestId":"a78f7426-d7b8-476b-b86f-504dbf771982","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_restore:failed_status[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0931f71a-5400-47c9-aed0-8fbfab5755d9] cannot determine cloud provider ","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=0931f71a-5400-47c9-aed0-8fbfab5755d9] cannot determine cloud provider ","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=0931f71a-5400-47c9-aed0-8fbfab5755d9] {"id":"66a13f7d-ebf8-4aa6-bc76-4e5763378ec1","timestamp":1706912525447,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3172de28-5168-4873-997c-c1dff965d835","backupName":"test-backup-restore-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","tenantType":"PAAS"}}}","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: [0m[33m[0.230ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-restore-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=0931f71a-5400-47c9-aed0-8fbfab5755d9] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=0931f71a-5400-47c9-aed0-8fbfab5755d9] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restore:failed status incompatible'.","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=0931f71a-5400-47c9-aed0-8fbfab5755d9] {"id":"1f776049-518a-4b85-88e7-bf7f38b28f27","timestamp":1706912525464,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3172de28-5168-4873-997c-c1dff965d835","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restore:failed status incompatible\"}],\"requestID\":\"0931f71a-5400-47c9-aed0-8fbfab5755d9\"}","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","tenantType":"PAAS"}}}","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=0931f71a-5400-47c9-aed0-8fbfab5755d9] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restore:failed status incompatible"}],"requestID":"0931f71a-5400-47c9-aed0-8fbfab5755d9"}
","requestId":"0931f71a-5400-47c9-aed0-8fbfab5755d9","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=10b996f3-e9c7-490f-96b4-e88e3de99444] cannot determine cloud provider ","requestId":"10b996f3-e9c7-490f-96b4-e88e3de99444","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=10b996f3-e9c7-490f-96b4-e88e3de99444] cannot determine cloud provider ","requestId":"10b996f3-e9c7-490f-96b4-e88e3de99444","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=10b996f3-e9c7-490f-96b4-e88e3de99444] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"10b996f3-e9c7-490f-96b4-e88e3de99444","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=10b996f3-e9c7-490f-96b4-e88e3de99444] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restore:warning status incompatible'.","requestId":"10b996f3-e9c7-490f-96b4-e88e3de99444","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=10b996f3-e9c7-490f-96b4-e88e3de99444] {"id":"433fa55e-f055-4858-9b43-6e971363789c","timestamp":1706912525611,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:afe1fd10-a00e-4479-9c84-dceeaa196e36","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restore:warning status incompatible\"}],\"requestID\":\"10b996f3-e9c7-490f-96b4-e88e3de99444\"}","requestId":"10b996f3-e9c7-490f-96b4-e88e3de99444","tenantType":"PAAS"}}}","requestId":"10b996f3-e9c7-490f-96b4-e88e3de99444","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=10b996f3-e9c7-490f-96b4-e88e3de99444] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restore:warning status incompatible"}],"requestID":"10b996f3-e9c7-490f-96b4-e88e3de99444"}
","requestId":"10b996f3-e9c7-490f-96b4-e88e3de99444","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=814d3e83-fe91-4909-a8ba-74ce39a19984] cannot determine cloud provider ","requestId":"814d3e83-fe91-4909-a8ba-74ce39a19984","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:22 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=814d3e83-fe91-4909-a8ba-74ce39a19984] cannot determine cloud provider ","requestId":"814d3e83-fe91-4909-a8ba-74ce39a19984","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=814d3e83-fe91-4909-a8ba-74ce39a19984] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"814d3e83-fe91-4909-a8ba-74ce39a19984","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=814d3e83-fe91-4909-a8ba-74ce39a19984] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. restore:succeeded status incompatible'.","requestId":"814d3e83-fe91-4909-a8ba-74ce39a19984","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=814d3e83-fe91-4909-a8ba-74ce39a19984] {"id":"b073f4e6-dd3d-4622-a0d0-47f5f8c7fde2","timestamp":1706912525762,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:5911ef2b-0c16-4716-b0b7-ae17a942335e","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restore:succeeded status incompatible\"}],\"requestID\":\"814d3e83-fe91-4909-a8ba-74ce39a19984\"}","requestId":"814d3e83-fe91-4909-a8ba-74ce39a19984","tenantType":"PAAS"}}}","requestId":"814d3e83-fe91-4909-a8ba-74ce39a19984","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=814d3e83-fe91-4909-a8ba-74ce39a19984] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restore:succeeded status incompatible"}],"requestID":"814d3e83-fe91-4909-a8ba-74ce39a19984"}
","requestId":"814d3e83-fe91-4909-a8ba-74ce39a19984","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e660e2ff-1e5c-47f7-a910-9e70c351d041] cannot determine cloud provider ","requestId":"e660e2ff-1e5c-47f7-a910-9e70c351d041","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e660e2ff-1e5c-47f7-a910-9e70c351d041] cannot determine cloud provider ","requestId":"e660e2ff-1e5c-47f7-a910-9e70c351d041","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=e660e2ff-1e5c-47f7-a910-9e70c351d041] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"e660e2ff-1e5c-47f7-a910-9e70c351d041","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=e660e2ff-1e5c-47f7-a910-9e70c351d041] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleDownForUpgrade:started status incompatible'.","requestId":"e660e2ff-1e5c-47f7-a910-9e70c351d041","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=e660e2ff-1e5c-47f7-a910-9e70c351d041] {"id":"dcd9b31c-7c4c-4832-b637-6769be8db766","timestamp":1706912525909,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d633ef23-5403-4985-a28f-70977bf1d3b4","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleDownForUpgrade:started status incompatible\"}],\"requestID\":\"e660e2ff-1e5c-47f7-a910-9e70c351d041\"}","requestId":"e660e2ff-1e5c-47f7-a910-9e70c351d041","tenantType":"PAAS"}}}","requestId":"e660e2ff-1e5c-47f7-a910-9e70c351d041","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=e660e2ff-1e5c-47f7-a910-9e70c351d041] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleDownForUpgrade:started status incompatible"}],"requestID":"e660e2ff-1e5c-47f7-a910-9e70c351d041"}
","requestId":"e660e2ff-1e5c-47f7-a910-9e70c351d041","timestamp":"2024-02-02T22:22:05Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=4a532a0b-32c2-40d2-9ed2-437fbf358ae7] cannot determine cloud provider ","requestId":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=4a532a0b-32c2-40d2-9ed2-437fbf358ae7] cannot determine cloud provider ","requestId":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=4a532a0b-32c2-40d2-9ed2-437fbf358ae7] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=4a532a0b-32c2-40d2-9ed2-437fbf358ae7] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleDownForUpgrade:finished status incompatible'.","requestId":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=4a532a0b-32c2-40d2-9ed2-437fbf358ae7] {"id":"350aeaf1-6659-4859-96e4-233c50f93961","timestamp":1706912526053,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:c782475a-144d-44c7-b82e-31115debec2a","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleDownForUpgrade:finished status incompatible\"}],\"requestID\":\"4a532a0b-32c2-40d2-9ed2-437fbf358ae7\"}","requestId":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7","tenantType":"PAAS"}}}","requestId":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=4a532a0b-32c2-40d2-9ed2-437fbf358ae7] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleDownForUpgrade:finished status incompatible"}],"requestID":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7"}
","requestId":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_scaleDownForUpgrade:failed_status[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8634ec44-aec0-48bc-88de-41a5567da7fa] cannot determine cloud provider ","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8634ec44-aec0-48bc-88de-41a5567da7fa] cannot determine cloud provider ","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=8634ec44-aec0-48bc-88de-41a5567da7fa] {"id":"cc1822bf-b1b2-476d-baa0-716376f807f7","timestamp":1706912526187,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:a5b5aff9-f291-4d1a-941f-5e71a7d1075e","backupName":"test-backup-scaleDownForUpgrade-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","tenantType":"PAAS"}}}","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: [0m[33m[0.298ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-scaleDownForUpgrade-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=8634ec44-aec0-48bc-88de-41a5567da7fa] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=8634ec44-aec0-48bc-88de-41a5567da7fa] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleDownForUpgrade:failed status incompatible'.","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=8634ec44-aec0-48bc-88de-41a5567da7fa] {"id":"52e4fc70-117f-4a88-af3e-f2d0c5fda3a1","timestamp":1706912526196,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:a5b5aff9-f291-4d1a-941f-5e71a7d1075e","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleDownForUpgrade:failed status incompatible\"}],\"requestID\":\"8634ec44-aec0-48bc-88de-41a5567da7fa\"}","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","tenantType":"PAAS"}}}","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=8634ec44-aec0-48bc-88de-41a5567da7fa] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleDownForUpgrade:failed status incompatible"}],"requestID":"8634ec44-aec0-48bc-88de-41a5567da7fa"}
","requestId":"8634ec44-aec0-48bc-88de-41a5567da7fa","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=fd6fa964-5fa4-47f1-98f1-3ab87346227f] cannot determine cloud provider ","requestId":"fd6fa964-5fa4-47f1-98f1-3ab87346227f","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=fd6fa964-5fa4-47f1-98f1-3ab87346227f] cannot determine cloud provider ","requestId":"fd6fa964-5fa4-47f1-98f1-3ab87346227f","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=fd6fa964-5fa4-47f1-98f1-3ab87346227f] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"fd6fa964-5fa4-47f1-98f1-3ab87346227f","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=fd6fa964-5fa4-47f1-98f1-3ab87346227f] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleDownForUpgrade:warning status incompatible'.","requestId":"fd6fa964-5fa4-47f1-98f1-3ab87346227f","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=fd6fa964-5fa4-47f1-98f1-3ab87346227f] {"id":"8cfa833f-f415-4ba7-85e1-6c4af9a390bd","timestamp":1706912526339,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:a69499cb-1565-454b-b3db-8ace99bf8219","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleDownForUpgrade:warning status incompatible\"}],\"requestID\":\"fd6fa964-5fa4-47f1-98f1-3ab87346227f\"}","requestId":"fd6fa964-5fa4-47f1-98f1-3ab87346227f","tenantType":"PAAS"}}}","requestId":"fd6fa964-5fa4-47f1-98f1-3ab87346227f","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=fd6fa964-5fa4-47f1-98f1-3ab87346227f] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleDownForUpgrade:warning status incompatible"}],"requestID":"fd6fa964-5fa4-47f1-98f1-3ab87346227f"}
","requestId":"fd6fa964-5fa4-47f1-98f1-3ab87346227f","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c1ec3c84-e974-4dc0-80bd-f5d31105f066] cannot determine cloud provider ","requestId":"c1ec3c84-e974-4dc0-80bd-f5d31105f066","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c1ec3c84-e974-4dc0-80bd-f5d31105f066] cannot determine cloud provider ","requestId":"c1ec3c84-e974-4dc0-80bd-f5d31105f066","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=c1ec3c84-e974-4dc0-80bd-f5d31105f066] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"c1ec3c84-e974-4dc0-80bd-f5d31105f066","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=c1ec3c84-e974-4dc0-80bd-f5d31105f066] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleDownForUpgrade:succeeded status incompatible'.","requestId":"c1ec3c84-e974-4dc0-80bd-f5d31105f066","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=c1ec3c84-e974-4dc0-80bd-f5d31105f066] {"id":"6c621c0e-d571-495d-9345-5ab9f1174c8c","timestamp":1706912526483,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:06f6b904-992e-48cf-b1ac-8b8a5aedc1f8","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleDownForUpgrade:succeeded status incompatible\"}],\"requestID\":\"c1ec3c84-e974-4dc0-80bd-f5d31105f066\"}","requestId":"c1ec3c84-e974-4dc0-80bd-f5d31105f066","tenantType":"PAAS"}}}","requestId":"c1ec3c84-e974-4dc0-80bd-f5d31105f066","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=c1ec3c84-e974-4dc0-80bd-f5d31105f066] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleDownForUpgrade:succeeded status incompatible"}],"requestID":"c1ec3c84-e974-4dc0-80bd-f5d31105f066"}
","requestId":"c1ec3c84-e974-4dc0-80bd-f5d31105f066","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8b23722c-7039-4e18-8528-1d5d74058a06] cannot determine cloud provider ","requestId":"8b23722c-7039-4e18-8528-1d5d74058a06","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=8b23722c-7039-4e18-8528-1d5d74058a06] cannot determine cloud provider ","requestId":"8b23722c-7039-4e18-8528-1d5d74058a06","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=8b23722c-7039-4e18-8528-1d5d74058a06] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"8b23722c-7039-4e18-8528-1d5d74058a06","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=8b23722c-7039-4e18-8528-1d5d74058a06] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleUpAfterUpgrade:started status incompatible'.","requestId":"8b23722c-7039-4e18-8528-1d5d74058a06","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=8b23722c-7039-4e18-8528-1d5d74058a06] {"id":"af3b734f-16f3-4a3c-b8d6-ea5b67b85408","timestamp":1706912526627,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3d52d4c7-217b-49c6-933c-9130bebae4ab","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleUpAfterUpgrade:started status incompatible\"}],\"requestID\":\"8b23722c-7039-4e18-8528-1d5d74058a06\"}","requestId":"8b23722c-7039-4e18-8528-1d5d74058a06","tenantType":"PAAS"}}}","requestId":"8b23722c-7039-4e18-8528-1d5d74058a06","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=8b23722c-7039-4e18-8528-1d5d74058a06] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleUpAfterUpgrade:started status incompatible"}],"requestID":"8b23722c-7039-4e18-8528-1d5d74058a06"}
","requestId":"8b23722c-7039-4e18-8528-1d5d74058a06","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=23bb9d00-348a-441c-9242-8575bb3156f8] cannot determine cloud provider ","requestId":"23bb9d00-348a-441c-9242-8575bb3156f8","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=23bb9d00-348a-441c-9242-8575bb3156f8] cannot determine cloud provider ","requestId":"23bb9d00-348a-441c-9242-8575bb3156f8","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=23bb9d00-348a-441c-9242-8575bb3156f8] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"23bb9d00-348a-441c-9242-8575bb3156f8","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=23bb9d00-348a-441c-9242-8575bb3156f8] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleUpAfterUpgrade:finished status incompatible'.","requestId":"23bb9d00-348a-441c-9242-8575bb3156f8","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=23bb9d00-348a-441c-9242-8575bb3156f8] {"id":"108e7fc3-7d47-4ef3-9eef-f4fa021eadcb","timestamp":1706912526772,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:1e789558-e2cf-4bbf-a43b-3ad262cd6443","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleUpAfterUpgrade:finished status incompatible\"}],\"requestID\":\"23bb9d00-348a-441c-9242-8575bb3156f8\"}","requestId":"23bb9d00-348a-441c-9242-8575bb3156f8","tenantType":"PAAS"}}}","requestId":"23bb9d00-348a-441c-9242-8575bb3156f8","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=23bb9d00-348a-441c-9242-8575bb3156f8] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleUpAfterUpgrade:finished status incompatible"}],"requestID":"23bb9d00-348a-441c-9242-8575bb3156f8"}
","requestId":"23bb9d00-348a-441c-9242-8575bb3156f8","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_scaleUpAfterUpgrade:failed_status[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=55876b62-758a-427d-923b-e8fc9d03b420] cannot determine cloud provider ","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=55876b62-758a-427d-923b-e8fc9d03b420] cannot determine cloud provider ","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=55876b62-758a-427d-923b-e8fc9d03b420] {"id":"d5b31798-14f7-4306-9a63-87e1c852c025","timestamp":1706912526908,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d65db27a-88f9-4d72-8fd3-82944b009b39","backupName":"test-backup-scaleUpAfterUpgrade-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","tenantType":"PAAS"}}}","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: [0m[33m[0.266ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-scaleUpAfterUpgrade-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=55876b62-758a-427d-923b-e8fc9d03b420] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=55876b62-758a-427d-923b-e8fc9d03b420] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleUpAfterUpgrade:failed status incompatible'.","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=55876b62-758a-427d-923b-e8fc9d03b420] {"id":"6435e3fb-13cd-44ee-9209-f84050f45428","timestamp":1706912526917,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d65db27a-88f9-4d72-8fd3-82944b009b39","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleUpAfterUpgrade:failed status incompatible\"}],\"requestID\":\"55876b62-758a-427d-923b-e8fc9d03b420\"}","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","tenantType":"PAAS"}}}","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=55876b62-758a-427d-923b-e8fc9d03b420] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleUpAfterUpgrade:failed status incompatible"}],"requestID":"55876b62-758a-427d-923b-e8fc9d03b420"}
","requestId":"55876b62-758a-427d-923b-e8fc9d03b420","timestamp":"2024-02-02T22:22:06Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cc6c4de7-483a-4ad4-90a6-85f7231bab34] cannot determine cloud provider ","requestId":"cc6c4de7-483a-4ad4-90a6-85f7231bab34","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cc6c4de7-483a-4ad4-90a6-85f7231bab34] cannot determine cloud provider ","requestId":"cc6c4de7-483a-4ad4-90a6-85f7231bab34","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cc6c4de7-483a-4ad4-90a6-85f7231bab34] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cc6c4de7-483a-4ad4-90a6-85f7231bab34","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=cc6c4de7-483a-4ad4-90a6-85f7231bab34] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleUpAfterUpgrade:warning status incompatible'.","requestId":"cc6c4de7-483a-4ad4-90a6-85f7231bab34","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cc6c4de7-483a-4ad4-90a6-85f7231bab34] {"id":"20e939b6-f0e9-4d32-92c0-ceb9333a2e5f","timestamp":1706912527071,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:ccc3da14-998a-4571-8f39-131f156b4a00","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleUpAfterUpgrade:warning status incompatible\"}],\"requestID\":\"cc6c4de7-483a-4ad4-90a6-85f7231bab34\"}","requestId":"cc6c4de7-483a-4ad4-90a6-85f7231bab34","tenantType":"PAAS"}}}","requestId":"cc6c4de7-483a-4ad4-90a6-85f7231bab34","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=cc6c4de7-483a-4ad4-90a6-85f7231bab34] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleUpAfterUpgrade:warning status incompatible"}],"requestID":"cc6c4de7-483a-4ad4-90a6-85f7231bab34"}
","requestId":"cc6c4de7-483a-4ad4-90a6-85f7231bab34","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=86e3aa93-2e14-479a-a82d-1f08c8a625b6] cannot determine cloud provider ","requestId":"86e3aa93-2e14-479a-a82d-1f08c8a625b6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=86e3aa93-2e14-479a-a82d-1f08c8a625b6] cannot determine cloud provider ","requestId":"86e3aa93-2e14-479a-a82d-1f08c8a625b6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=86e3aa93-2e14-479a-a82d-1f08c8a625b6] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"86e3aa93-2e14-479a-a82d-1f08c8a625b6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=86e3aa93-2e14-479a-a82d-1f08c8a625b6] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. scaleUpAfterUpgrade:succeeded status incompatible'.","requestId":"86e3aa93-2e14-479a-a82d-1f08c8a625b6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=86e3aa93-2e14-479a-a82d-1f08c8a625b6] {"id":"d8baff6c-07cc-4573-91c7-d1a6faff14f0","timestamp":1706912527216,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:8bbd38dc-73d0-4ddb-9ba8-3aa28025e581","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. scaleUpAfterUpgrade:succeeded status incompatible\"}],\"requestID\":\"86e3aa93-2e14-479a-a82d-1f08c8a625b6\"}","requestId":"86e3aa93-2e14-479a-a82d-1f08c8a625b6","tenantType":"PAAS"}}}","requestId":"86e3aa93-2e14-479a-a82d-1f08c8a625b6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=86e3aa93-2e14-479a-a82d-1f08c8a625b6] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. scaleUpAfterUpgrade:succeeded status incompatible"}],"requestID":"86e3aa93-2e14-479a-a82d-1f08c8a625b6"}
","requestId":"86e3aa93-2e14-479a-a82d-1f08c8a625b6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=f14dd939-a8c7-4a5b-94dd-4cd13320322a] cannot determine cloud provider ","requestId":"f14dd939-a8c7-4a5b-94dd-4cd13320322a","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=f14dd939-a8c7-4a5b-94dd-4cd13320322a] cannot determine cloud provider ","requestId":"f14dd939-a8c7-4a5b-94dd-4cd13320322a","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=f14dd939-a8c7-4a5b-94dd-4cd13320322a] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"f14dd939-a8c7-4a5b-94dd-4cd13320322a","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=f14dd939-a8c7-4a5b-94dd-4cd13320322a] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migration:started status incompatible'.","requestId":"f14dd939-a8c7-4a5b-94dd-4cd13320322a","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=f14dd939-a8c7-4a5b-94dd-4cd13320322a] {"id":"86d597e4-fc18-4cea-a351-3bfd35abaf50","timestamp":1706912527368,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:7374e87d-84af-4ecd-9895-b6edce582dc0","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migration:started status incompatible\"}],\"requestID\":\"f14dd939-a8c7-4a5b-94dd-4cd13320322a\"}","requestId":"f14dd939-a8c7-4a5b-94dd-4cd13320322a","tenantType":"PAAS"}}}","requestId":"f14dd939-a8c7-4a5b-94dd-4cd13320322a","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=f14dd939-a8c7-4a5b-94dd-4cd13320322a] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migration:started status incompatible"}],"requestID":"f14dd939-a8c7-4a5b-94dd-4cd13320322a"}
","requestId":"f14dd939-a8c7-4a5b-94dd-4cd13320322a","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=5555c047-1198-4654-b423-2ba3f2e5b65e] cannot determine cloud provider ","requestId":"5555c047-1198-4654-b423-2ba3f2e5b65e","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=5555c047-1198-4654-b423-2ba3f2e5b65e] cannot determine cloud provider ","requestId":"5555c047-1198-4654-b423-2ba3f2e5b65e","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=5555c047-1198-4654-b423-2ba3f2e5b65e] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"5555c047-1198-4654-b423-2ba3f2e5b65e","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=5555c047-1198-4654-b423-2ba3f2e5b65e] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migration:finished status incompatible'.","requestId":"5555c047-1198-4654-b423-2ba3f2e5b65e","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=5555c047-1198-4654-b423-2ba3f2e5b65e] {"id":"ed29f2d6-f4f4-49fd-b8f9-f20097d1c26d","timestamp":1706912527516,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:eab5f27e-89f3-4ce5-a6a9-362a6c55ac0f","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migration:finished status incompatible\"}],\"requestID\":\"5555c047-1198-4654-b423-2ba3f2e5b65e\"}","requestId":"5555c047-1198-4654-b423-2ba3f2e5b65e","tenantType":"PAAS"}}}","requestId":"5555c047-1198-4654-b423-2ba3f2e5b65e","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=5555c047-1198-4654-b423-2ba3f2e5b65e] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migration:finished status incompatible"}],"requestID":"5555c047-1198-4654-b423-2ba3f2e5b65e"}
","requestId":"5555c047-1198-4654-b423-2ba3f2e5b65e","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_migration:failed_status[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c7837baa-ec73-46af-a7bf-1580fb37d3d6] cannot determine cloud provider ","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c7837baa-ec73-46af-a7bf-1580fb37d3d6] cannot determine cloud provider ","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=c7837baa-ec73-46af-a7bf-1580fb37d3d6] {"id":"371678b9-9a2b-452f-a303-0a68a5a802bf","timestamp":1706912527658,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:663879c8-60f9-4cd5-970d-7fa62c2f76e4","backupName":"test-backup-migration-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","tenantType":"PAAS"}}}","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: [0m[33m[0.238ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-migration-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=c7837baa-ec73-46af-a7bf-1580fb37d3d6] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=c7837baa-ec73-46af-a7bf-1580fb37d3d6] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migration:failed status incompatible'.","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=c7837baa-ec73-46af-a7bf-1580fb37d3d6] {"id":"8fd2baee-34ed-4f94-ab91-6cd47ffb7c70","timestamp":1706912527670,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:663879c8-60f9-4cd5-970d-7fa62c2f76e4","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migration:failed status incompatible\"}],\"requestID\":\"c7837baa-ec73-46af-a7bf-1580fb37d3d6\"}","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","tenantType":"PAAS"}}}","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=c7837baa-ec73-46af-a7bf-1580fb37d3d6] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migration:failed status incompatible"}],"requestID":"c7837baa-ec73-46af-a7bf-1580fb37d3d6"}
","requestId":"c7837baa-ec73-46af-a7bf-1580fb37d3d6","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e05e53de-9a1b-4add-a456-a2348e6a2743] cannot determine cloud provider ","requestId":"e05e53de-9a1b-4add-a456-a2348e6a2743","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=e05e53de-9a1b-4add-a456-a2348e6a2743] cannot determine cloud provider ","requestId":"e05e53de-9a1b-4add-a456-a2348e6a2743","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=e05e53de-9a1b-4add-a456-a2348e6a2743] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"e05e53de-9a1b-4add-a456-a2348e6a2743","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=e05e53de-9a1b-4add-a456-a2348e6a2743] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migration:warning status incompatible'.","requestId":"e05e53de-9a1b-4add-a456-a2348e6a2743","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=e05e53de-9a1b-4add-a456-a2348e6a2743] {"id":"2a8c6144-72cb-4977-8d54-290b40f3d660","timestamp":1706912527819,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:941d9c92-afa8-42e5-bf7a-cb54c7183555","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migration:warning status incompatible\"}],\"requestID\":\"e05e53de-9a1b-4add-a456-a2348e6a2743\"}","requestId":"e05e53de-9a1b-4add-a456-a2348e6a2743","tenantType":"PAAS"}}}","requestId":"e05e53de-9a1b-4add-a456-a2348e6a2743","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=e05e53de-9a1b-4add-a456-a2348e6a2743] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migration:warning status incompatible"}],"requestID":"e05e53de-9a1b-4add-a456-a2348e6a2743"}
","requestId":"e05e53de-9a1b-4add-a456-a2348e6a2743","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=14fb8a30-08c8-4522-883f-f21ad3d1f385] cannot determine cloud provider ","requestId":"14fb8a30-08c8-4522-883f-f21ad3d1f385","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=14fb8a30-08c8-4522-883f-f21ad3d1f385] cannot determine cloud provider ","requestId":"14fb8a30-08c8-4522-883f-f21ad3d1f385","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=14fb8a30-08c8-4522-883f-f21ad3d1f385] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"14fb8a30-08c8-4522-883f-f21ad3d1f385","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=14fb8a30-08c8-4522-883f-f21ad3d1f385] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migration:succeeded status incompatible'.","requestId":"14fb8a30-08c8-4522-883f-f21ad3d1f385","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=14fb8a30-08c8-4522-883f-f21ad3d1f385] {"id":"184421ba-9e14-497a-8869-ea3feca7238c","timestamp":1706912527977,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:abe1ce13-2f58-4b0a-9086-e61fabb4c716","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migration:succeeded status incompatible\"}],\"requestID\":\"14fb8a30-08c8-4522-883f-f21ad3d1f385\"}","requestId":"14fb8a30-08c8-4522-883f-f21ad3d1f385","tenantType":"PAAS"}}}","requestId":"14fb8a30-08c8-4522-883f-f21ad3d1f385","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=14fb8a30-08c8-4522-883f-f21ad3d1f385] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migration:succeeded status incompatible"}],"requestID":"14fb8a30-08c8-4522-883f-f21ad3d1f385"}
","requestId":"14fb8a30-08c8-4522-883f-f21ad3d1f385","timestamp":"2024-02-02T22:22:07Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=6997e506-18e5-450d-ba00-c6eab87c1fa6] cannot determine cloud provider ","requestId":"6997e506-18e5-450d-ba00-c6eab87c1fa6","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=6997e506-18e5-450d-ba00-c6eab87c1fa6] cannot determine cloud provider ","requestId":"6997e506-18e5-450d-ba00-c6eab87c1fa6","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=6997e506-18e5-450d-ba00-c6eab87c1fa6] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"6997e506-18e5-450d-ba00-c6eab87c1fa6","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=6997e506-18e5-450d-ba00-c6eab87c1fa6] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migrationReadinessCheck:started status incompatible'.","requestId":"6997e506-18e5-450d-ba00-c6eab87c1fa6","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=6997e506-18e5-450d-ba00-c6eab87c1fa6] {"id":"b1a0366b-45cc-4004-b100-5a41073a770e","timestamp":1706912528136,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:76e0d600-ac4f-4c43-b301-59e0cec9486d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migrationReadinessCheck:started status incompatible\"}],\"requestID\":\"6997e506-18e5-450d-ba00-c6eab87c1fa6\"}","requestId":"6997e506-18e5-450d-ba00-c6eab87c1fa6","tenantType":"PAAS"}}}","requestId":"6997e506-18e5-450d-ba00-c6eab87c1fa6","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=6997e506-18e5-450d-ba00-c6eab87c1fa6] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migrationReadinessCheck:started status incompatible"}],"requestID":"6997e506-18e5-450d-ba00-c6eab87c1fa6"}
","requestId":"6997e506-18e5-450d-ba00-c6eab87c1fa6","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9] cannot determine cloud provider ","requestId":"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9] cannot determine cloud provider ","requestId":"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migrationReadinessCheck:finished status incompatible'.","requestId":"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9] {"id":"4b2bdb0a-4e62-43ff-aba1-00a680b2de35","timestamp":1706912528294,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:89cfb6dd-b850-42c0-93de-06ce1ae9f883","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migrationReadinessCheck:finished status incompatible\"}],\"requestID\":\"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9\"}","requestId":"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9","tenantType":"PAAS"}}}","requestId":"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migrationReadinessCheck:finished status incompatible"}],"requestID":"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9"}
","requestId":"d5c6c7b7-92b6-45c2-b8d6-5f3d460c3cb9","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_migrationReadinessCheck:failed_status[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cb4507a9-a3df-4e88-97ff-fb0e3e132beb] cannot determine cloud provider ","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cb4507a9-a3df-4e88-97ff-fb0e3e132beb] cannot determine cloud provider ","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cb4507a9-a3df-4e88-97ff-fb0e3e132beb] {"id":"b479ca5e-6a63-4ef5-b726-f44b5b638e44","timestamp":1706912528433,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:83170b98-2e97-408f-81f2-c6a548bdc01e","backupName":"test-backup-migrationReadinessCheck-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","tenantType":"PAAS"}}}","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: [0m[33m[0.278ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-migrationReadinessCheck-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cb4507a9-a3df-4e88-97ff-fb0e3e132beb] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=cb4507a9-a3df-4e88-97ff-fb0e3e132beb] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migrationReadinessCheck:failed status incompatible'.","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cb4507a9-a3df-4e88-97ff-fb0e3e132beb] {"id":"372a1981-2d55-40c9-9c08-b169e962d466","timestamp":1706912528445,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:83170b98-2e97-408f-81f2-c6a548bdc01e","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migrationReadinessCheck:failed status incompatible\"}],\"requestID\":\"cb4507a9-a3df-4e88-97ff-fb0e3e132beb\"}","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","tenantType":"PAAS"}}}","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=cb4507a9-a3df-4e88-97ff-fb0e3e132beb] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migrationReadinessCheck:failed status incompatible"}],"requestID":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb"}
","requestId":"cb4507a9-a3df-4e88-97ff-fb0e3e132beb","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=b462beaf-1467-40d9-bca9-936d04bd9362] cannot determine cloud provider ","requestId":"b462beaf-1467-40d9-bca9-936d04bd9362","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=b462beaf-1467-40d9-bca9-936d04bd9362] cannot determine cloud provider ","requestId":"b462beaf-1467-40d9-bca9-936d04bd9362","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=b462beaf-1467-40d9-bca9-936d04bd9362] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"b462beaf-1467-40d9-bca9-936d04bd9362","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=b462beaf-1467-40d9-bca9-936d04bd9362] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migrationReadinessCheck:warning status incompatible'.","requestId":"b462beaf-1467-40d9-bca9-936d04bd9362","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=b462beaf-1467-40d9-bca9-936d04bd9362] {"id":"84e9b7f0-b220-4db4-a325-a140115db382","timestamp":1706912528594,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:55115d00-1691-4056-beac-acda1e5c5bbc","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migrationReadinessCheck:warning status incompatible\"}],\"requestID\":\"b462beaf-1467-40d9-bca9-936d04bd9362\"}","requestId":"b462beaf-1467-40d9-bca9-936d04bd9362","tenantType":"PAAS"}}}","requestId":"b462beaf-1467-40d9-bca9-936d04bd9362","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=b462beaf-1467-40d9-bca9-936d04bd9362] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migrationReadinessCheck:warning status incompatible"}],"requestID":"b462beaf-1467-40d9-bca9-936d04bd9362"}
","requestId":"b462beaf-1467-40d9-bca9-936d04bd9362","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=add4904c-8287-43ec-a941-8ec8090452e3] cannot determine cloud provider ","requestId":"add4904c-8287-43ec-a941-8ec8090452e3","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=add4904c-8287-43ec-a941-8ec8090452e3] cannot determine cloud provider ","requestId":"add4904c-8287-43ec-a941-8ec8090452e3","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=add4904c-8287-43ec-a941-8ec8090452e3] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"add4904c-8287-43ec-a941-8ec8090452e3","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=add4904c-8287-43ec-a941-8ec8090452e3] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. migrationReadinessCheck:succeeded status incompatible'.","requestId":"add4904c-8287-43ec-a941-8ec8090452e3","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=add4904c-8287-43ec-a941-8ec8090452e3] {"id":"d92cde9a-2ea0-466d-b884-3b3e825c4213","timestamp":1706912528759,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:de020f80-8d1a-45ec-8e50-e653135542af","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. migrationReadinessCheck:succeeded status incompatible\"}],\"requestID\":\"add4904c-8287-43ec-a941-8ec8090452e3\"}","requestId":"add4904c-8287-43ec-a941-8ec8090452e3","tenantType":"PAAS"}}}","requestId":"add4904c-8287-43ec-a941-8ec8090452e3","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=add4904c-8287-43ec-a941-8ec8090452e3] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. migrationReadinessCheck:succeeded status incompatible"}],"requestID":"add4904c-8287-43ec-a941-8ec8090452e3"}
","requestId":"add4904c-8287-43ec-a941-8ec8090452e3","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=009ef640-739c-424a-a45c-c106b5512df7] cannot determine cloud provider ","requestId":"009ef640-739c-424a-a45c-c106b5512df7","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=009ef640-739c-424a-a45c-c106b5512df7] cannot determine cloud provider ","requestId":"009ef640-739c-424a-a45c-c106b5512df7","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=009ef640-739c-424a-a45c-c106b5512df7] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"009ef640-739c-424a-a45c-c106b5512df7","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=009ef640-739c-424a-a45c-c106b5512df7] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. validateMigration:started status incompatible'.","requestId":"009ef640-739c-424a-a45c-c106b5512df7","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=009ef640-739c-424a-a45c-c106b5512df7] {"id":"df27b329-6bdc-4206-82de-b26f869c2879","timestamp":1706912528908,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:e5381411-1d54-4c6b-ba47-4eb9c1159e19","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. validateMigration:started status incompatible\"}],\"requestID\":\"009ef640-739c-424a-a45c-c106b5512df7\"}","requestId":"009ef640-739c-424a-a45c-c106b5512df7","tenantType":"PAAS"}}}","requestId":"009ef640-739c-424a-a45c-c106b5512df7","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=009ef640-739c-424a-a45c-c106b5512df7] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. validateMigration:started status incompatible"}],"requestID":"009ef640-739c-424a-a45c-c106b5512df7"}
","requestId":"009ef640-739c-424a-a45c-c106b5512df7","timestamp":"2024-02-02T22:22:08Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c7dd47c1-2992-4730-b3c6-a414cda80b11] cannot determine cloud provider ","requestId":"c7dd47c1-2992-4730-b3c6-a414cda80b11","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=c7dd47c1-2992-4730-b3c6-a414cda80b11] cannot determine cloud provider ","requestId":"c7dd47c1-2992-4730-b3c6-a414cda80b11","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=c7dd47c1-2992-4730-b3c6-a414cda80b11] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"c7dd47c1-2992-4730-b3c6-a414cda80b11","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=c7dd47c1-2992-4730-b3c6-a414cda80b11] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. validateMigration:finished status incompatible'.","requestId":"c7dd47c1-2992-4730-b3c6-a414cda80b11","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=c7dd47c1-2992-4730-b3c6-a414cda80b11] {"id":"7a892df4-3b24-423f-b9a4-78a78d97dcaa","timestamp":1706912529070,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:712ad585-5436-4c6e-a842-6ccb51ca55df","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. validateMigration:finished status incompatible\"}],\"requestID\":\"c7dd47c1-2992-4730-b3c6-a414cda80b11\"}","requestId":"c7dd47c1-2992-4730-b3c6-a414cda80b11","tenantType":"PAAS"}}}","requestId":"c7dd47c1-2992-4730-b3c6-a414cda80b11","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=c7dd47c1-2992-4730-b3c6-a414cda80b11] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. validateMigration:finished status incompatible"}],"requestID":"c7dd47c1-2992-4730-b3c6-a414cda80b11"}
","requestId":"c7dd47c1-2992-4730-b3c6-a414cda80b11","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_validateMigration:failed_status[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=94d179d8-2ea7-4207-ad45-8969eae6d64f] cannot determine cloud provider ","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=94d179d8-2ea7-4207-ad45-8969eae6d64f] cannot determine cloud provider ","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=94d179d8-2ea7-4207-ad45-8969eae6d64f] {"id":"93cb1a88-94d8-4972-941b-8d5810fa2796","timestamp":1706912529215,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:a90e9d0a-a543-456c-8a48-9d7d3444321d","backupName":"test-backup-validateMigration-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","tenantType":"PAAS"}}}","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: [0m[33m[0.226ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-validateMigration-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=94d179d8-2ea7-4207-ad45-8969eae6d64f] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=94d179d8-2ea7-4207-ad45-8969eae6d64f] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. validateMigration:failed status incompatible'.","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=94d179d8-2ea7-4207-ad45-8969eae6d64f] {"id":"01dab503-07ac-46d9-95bc-b2e5244bad5d","timestamp":1706912529224,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:a90e9d0a-a543-456c-8a48-9d7d3444321d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. validateMigration:failed status incompatible\"}],\"requestID\":\"94d179d8-2ea7-4207-ad45-8969eae6d64f\"}","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","tenantType":"PAAS"}}}","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=94d179d8-2ea7-4207-ad45-8969eae6d64f] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. validateMigration:failed status incompatible"}],"requestID":"94d179d8-2ea7-4207-ad45-8969eae6d64f"}
","requestId":"94d179d8-2ea7-4207-ad45-8969eae6d64f","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=20353f0a-a3c5-4186-a1c1-edc29f20110a] cannot determine cloud provider ","requestId":"20353f0a-a3c5-4186-a1c1-edc29f20110a","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=20353f0a-a3c5-4186-a1c1-edc29f20110a] cannot determine cloud provider ","requestId":"20353f0a-a3c5-4186-a1c1-edc29f20110a","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=20353f0a-a3c5-4186-a1c1-edc29f20110a] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"20353f0a-a3c5-4186-a1c1-edc29f20110a","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=20353f0a-a3c5-4186-a1c1-edc29f20110a] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. validateMigration:warning status incompatible'.","requestId":"20353f0a-a3c5-4186-a1c1-edc29f20110a","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=20353f0a-a3c5-4186-a1c1-edc29f20110a] {"id":"d4983059-d4c7-4f9f-b849-aa7014a6df02","timestamp":1706912529371,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:69a1fadd-314c-4ae5-b773-9763896bb8a7","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. validateMigration:warning status incompatible\"}],\"requestID\":\"20353f0a-a3c5-4186-a1c1-edc29f20110a\"}","requestId":"20353f0a-a3c5-4186-a1c1-edc29f20110a","tenantType":"PAAS"}}}","requestId":"20353f0a-a3c5-4186-a1c1-edc29f20110a","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=20353f0a-a3c5-4186-a1c1-edc29f20110a] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. validateMigration:warning status incompatible"}],"requestID":"20353f0a-a3c5-4186-a1c1-edc29f20110a"}
","requestId":"20353f0a-a3c5-4186-a1c1-edc29f20110a","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a16adb06-792c-4ef5-b296-b71bdc49b5fa] cannot determine cloud provider ","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a16adb06-792c-4ef5-b296-b71bdc49b5fa] cannot determine cloud provider ","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a16adb06-792c-4ef5-b296-b71bdc49b5fa] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=a16adb06-792c-4ef5-b296-b71bdc49b5fa] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. validateMigration:succeeded status incompatible'.","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a16adb06-792c-4ef5-b296-b71bdc49b5fa] {"id":"3412a7c9-96a6-4a71-9e36-1aff5a28bc83","timestamp":1706912529522,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:0118144f-4c2d-4734-9edb-6977b45464af","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. validateMigration:succeeded status incompatible\"}],\"requestID\":\"a16adb06-792c-4ef5-b296-b71bdc49b5fa\"}","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","tenantType":"PAAS"}}}","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=a16adb06-792c-4ef5-b296-b71bdc49b5fa] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. validateMigration:succeeded status incompatible"}],"requestID":"a16adb06-792c-4ef5-b296-b71bdc49b5fa"}
","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cd3b6929-ed0c-48bf-b25e-48fadcf40c5b] cannot determine cloud provider ","requestId":"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cd3b6929-ed0c-48bf-b25e-48fadcf40c5b] cannot determine cloud provider ","requestId":"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cd3b6929-ed0c-48bf-b25e-48fadcf40c5b] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=cd3b6929-ed0c-48bf-b25e-48fadcf40c5b] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. connection:started status incompatible'.","requestId":"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cd3b6929-ed0c-48bf-b25e-48fadcf40c5b] {"id":"a7d6e11b-3dbb-44a7-99a8-afe924d65cce","timestamp":1706912529700,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:4e7a0765-2bd5-443f-afd3-d40d62feeabf","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. connection:started status incompatible\"}],\"requestID\":\"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b\"}","requestId":"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b","tenantType":"PAAS"}}}","requestId":"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=cd3b6929-ed0c-48bf-b25e-48fadcf40c5b] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. connection:started status incompatible"}],"requestID":"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b"}
","requestId":"cd3b6929-ed0c-48bf-b25e-48fadcf40c5b","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d] cannot determine cloud provider ","requestId":"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:23 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d] cannot determine cloud provider ","requestId":"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. connection:finished status incompatible'.","requestId":"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d] {"id":"54a3123e-7cc1-4c8a-8c79-04a66df00e33","timestamp":1706912529850,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:d28575cf-4912-4b80-833d-65da8b3e2f1f","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. connection:finished status incompatible\"}],\"requestID\":\"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d\"}","requestId":"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d","tenantType":"PAAS"}}}","requestId":"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. connection:finished status incompatible"}],"requestID":"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d"}
","requestId":"cf1ce9c3-0f18-4a5f-a01e-307a8c5f263d","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: === RUN   TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_connection:failed_status[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a4ae9ff1-3ae1-4307-bbd9-deabf19c3929] cannot determine cloud provider ","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","timestamp":"2024-02-02T22:22:09Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a4ae9ff1-3ae1-4307-bbd9-deabf19c3929] cannot determine cloud provider ","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a4ae9ff1-3ae1-4307-bbd9-deabf19c3929] {"id":"9767f3e8-d6f0-4760-8368-66c5edca5b51","timestamp":1706912530001,"Details":{"CdpCMLWorkspaceBackupRequested":{"workspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:8b7d2e93-0560-45ad-880a-6839bb492b50","backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:476e6cc3-34a0-4ead-bf50-b4ddc55f22e7","backupName":"test-backup-connection-failed","accountId":"testAccountID","environmentType":1,"environmentCrn":"crn:altus:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:93c95d92-5a73-4f58-ac26-3fcfc09b2523","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","tenantType":"PAAS"}}}","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: [0m[33m[0.402ms] [34;1m[rows:0][0m SELECT * FROM `backup_catalog` WHERE `backup_catalog`.`workspace_id` = 1 AND `backup_catalog`.`backup_name` = "test-backup-connection-failed" AND deleted_at is NULL ORDER BY `backup_catalog`.`id` LIMIT 1[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a4ae9ff1-3ae1-4307-bbd9-deabf19c3929] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=a4ae9ff1-3ae1-4307-bbd9-deabf19c3929] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. connection:failed status incompatible'.","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a4ae9ff1-3ae1-4307-bbd9-deabf19c3929] {"id":"62d623ce-0dd3-44a0-a2d4-0bba99976891","timestamp":1706912530013,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:476e6cc3-34a0-4ead-bf50-b4ddc55f22e7","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. connection:failed status incompatible\"}],\"requestID\":\"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929\"}","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","tenantType":"PAAS"}}}","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=a4ae9ff1-3ae1-4307-bbd9-deabf19c3929] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. connection:failed status incompatible"}],"requestID":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929"}
","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=af55e947-f852-4f56-ac72-536e586059e0] cannot determine cloud provider ","requestId":"af55e947-f852-4f56-ac72-536e586059e0","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=af55e947-f852-4f56-ac72-536e586059e0] cannot determine cloud provider ","requestId":"af55e947-f852-4f56-ac72-536e586059e0","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=af55e947-f852-4f56-ac72-536e586059e0] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"af55e947-f852-4f56-ac72-536e586059e0","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=af55e947-f852-4f56-ac72-536e586059e0] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. connection:warning status incompatible'.","requestId":"af55e947-f852-4f56-ac72-536e586059e0","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=af55e947-f852-4f56-ac72-536e586059e0] {"id":"e5392efb-b6e8-4654-8635-8ce2d42ab3c1","timestamp":1706912530184,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:7c86185f-6187-49fd-9368-905bb8209796","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. connection:warning status incompatible\"}],\"requestID\":\"af55e947-f852-4f56-ac72-536e586059e0\"}","requestId":"af55e947-f852-4f56-ac72-536e586059e0","tenantType":"PAAS"}}}","requestId":"af55e947-f852-4f56-ac72-536e586059e0","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=af55e947-f852-4f56-ac72-536e586059e0] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. connection:warning status incompatible"}],"requestID":"af55e947-f852-4f56-ac72-536e586059e0"}
","requestId":"af55e947-f852-4f56-ac72-536e586059e0","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cae8d956-1513-4d18-b17f-e4dd945b80cf] cannot determine cloud provider ","requestId":"cae8d956-1513-4d18-b17f-e4dd945b80cf","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=cae8d956-1513-4d18-b17f-e4dd945b80cf] cannot determine cloud provider ","requestId":"cae8d956-1513-4d18-b17f-e4dd945b80cf","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cae8d956-1513-4d18-b17f-e4dd945b80cf] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cae8d956-1513-4d18-b17f-e4dd945b80cf","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=cae8d956-1513-4d18-b17f-e4dd945b80cf] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. connection:succeeded status incompatible'.","requestId":"cae8d956-1513-4d18-b17f-e4dd945b80cf","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cae8d956-1513-4d18-b17f-e4dd945b80cf] {"id":"48cd85b5-db4d-4211-aac6-b973f1ccfe2f","timestamp":1706912530346,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:6cc702f2-95e7-459f-b1c5-9f369f492281","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. connection:succeeded status incompatible\"}],\"requestID\":\"cae8d956-1513-4d18-b17f-e4dd945b80cf\"}","requestId":"cae8d956-1513-4d18-b17f-e4dd945b80cf","tenantType":"PAAS"}}}","requestId":"cae8d956-1513-4d18-b17f-e4dd945b80cf","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=cae8d956-1513-4d18-b17f-e4dd945b80cf] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. connection:succeeded status incompatible"}],"requestID":"cae8d956-1513-4d18-b17f-e4dd945b80cf"}
","requestId":"cae8d956-1513-4d18-b17f-e4dd945b80cf","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_provision:failed_status (0.13s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_deprovision:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_installation:failed_status (0.16s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_mount:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_renew:failed_status (0.15s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_suspend:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_resume:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_upgrade:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_modify:failed_status (0.15s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_unmount:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_provisionRequestValidation:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_preInstallValidation:failed_status (0.15s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_shutDownForBackup:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_restartAfterBackup:failed_status (0.16s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_backup:failed_status (0.33s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_restore:failed_status (0.15s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_scaleDownForUpgrade:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_scaleUpAfterUpgrade:failed_status (0.14s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_migration:failed_status (0.15s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_migrationReadinessCheck:failed_status (0.15s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_validateMigration:failed_status (0.15s)[0m
2024-02-02 22:22:24 - INFO-root::util|431::         --- PASS: TestBackup/Test_validator_for_backupable_statuses/workspace_backup_for_connection:failed_status (0.16s)[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=a3963689-8198-4632-b689-9d3a6a3444a8] cannot determine cloud provider ","requestId":"a3963689-8198-4632-b689-9d3a6a3444a8","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=3d709ee9-acf0-4c0a-8bf9-65db396977e6] could not obtain default workload version from configuration variables in DB","requestId":"3d709ee9-acf0-4c0a-8bf9-65db396977e6","timestamp":"2024-02-02T22:22:10Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=152501ee-741b-4464-a3fa-56d26a99929c] cannot determine cloud provider ","requestId":"152501ee-741b-4464-a3fa-56d26a99929c","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:821","message":"[requestId=64a60bd6-1242-4de5-901e-436238bb38a5] Cloud API controller not found for platform: STANDARD","requestId":"64a60bd6-1242-4de5-901e-436238bb38a5","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"dynamicaddon/dynamicaddon.go:176","message":"[requestId=64a60bd6-1242-4de5-901e-436238bb38a5] Failed to lookup Spark addons: Get "/api/v1/latestComponentInfo": unsupported protocol scheme ""","requestId":"64a60bd6-1242-4de5-901e-436238bb38a5","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=72d39dd7-931e-4074-8b23-6acaa8c6981f] cannot determine cloud provider ","requestId":"72d39dd7-931e-4074-8b23-6acaa8c6981f","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=a2257e46-773c-4b18-88d6-3c3010495394] could not obtain default workload version from configuration variables in DB","requestId":"a2257e46-773c-4b18-88d6-3c3010495394","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"validator/validator.go:1850","message":"[requestId=a2257e46-773c-4b18-88d6-3c3010495394] backup snapshot validation error: random validation error","requestId":"a2257e46-773c-4b18-88d6-3c3010495394","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=a2257e46-773c-4b18-88d6-3c3010495394] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"a2257e46-773c-4b18-88d6-3c3010495394","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=a2257e46-773c-4b18-88d6-3c3010495394] Pre-flight validation Category: 'FAILED', Name: 'Backup Snapshot Validation', Message: 'Some/All resources related to backup snapshot were missing'.","requestId":"a2257e46-773c-4b18-88d6-3c3010495394","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a2257e46-773c-4b18-88d6-3c3010495394] {"id":"b083f46b-242d-4e80-894d-372dad5064c3","timestamp":1706912531539,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:997f6d2c-de45-4974-9430-bd3e02338156","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Backup Snapshot Validation\",\"detailedMessage\":\"Some/All resources related to backup snapshot were missing. Error: random validation error\"}],\"requestID\":\"a2257e46-773c-4b18-88d6-3c3010495394\"}","requestId":"a2257e46-773c-4b18-88d6-3c3010495394","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:316e3635-d6ba-4efd-8976-a13026357470"},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"a2257e46-773c-4b18-88d6-3c3010495394","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=a2257e46-773c-4b18-88d6-3c3010495394] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:997f6d2c-de45-4974-9430-bd3e02338156"}'","requestId":"a2257e46-773c-4b18-88d6-3c3010495394","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:187","message":"[requestId=a2257e46-773c-4b18-88d6-3c3010495394] failed to start RestoreWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Backup Snapshot Validation","detailedMessage":"Some/All resources related to backup snapshot were missing. Error: random validation error"}],"requestID":"a2257e46-773c-4b18-88d6-3c3010495394"}
","requestId":"a2257e46-773c-4b18-88d6-3c3010495394","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=abbad230-324d-474a-8c79-ad38dc9e0d78] cannot determine cloud provider ","requestId":"abbad230-324d-474a-8c79-ad38dc9e0d78","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=025fe01c-a01e-423e-b9df-966ac5a0a93b] could not obtain default workload version from configuration variables in DB","requestId":"025fe01c-a01e-423e-b9df-966ac5a0a93b","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=025fe01c-a01e-423e-b9df-966ac5a0a93b] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"025fe01c-a01e-423e-b9df-966ac5a0a93b","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=025fe01c-a01e-423e-b9df-966ac5a0a93b] {"id":"debab280-6300-4f92-a947-b32a3cf096b9","timestamp":1706912531817,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:testAccountID:workspace:a9988049-8460-4fee-b73d-411b3f70945a","oldStatus":23,"newStatus":25,"failureReason":"{\"failedValidations\":[{\"category\":\"CML_NFS\",\"name\":\"Mock validator\",\"detailedMessage\":\"Mock validation fails for restore workspace\"}],\"requestID\":\"025fe01c-a01e-423e-b9df-966ac5a0a93b\"}","requestId":"025fe01c-a01e-423e-b9df-966ac5a0a93b","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:e20d5990-2bad-4a41-aac1-84a00bb6adde"},"tenantType":"PAAS","updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"025fe01c-a01e-423e-b9df-966ac5a0a93b","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=025fe01c-a01e-423e-b9df-966ac5a0a93b] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace:a9988049-8460-4fee-b73d-411b3f70945a"}'","requestId":"025fe01c-a01e-423e-b9df-966ac5a0a93b","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:187","message":"[requestId=025fe01c-a01e-423e-b9df-966ac5a0a93b] failed to start RestoreWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"CML_NFS","name":"Mock validator","detailedMessage":"Mock validation fails for restore workspace"}],"requestID":"025fe01c-a01e-423e-b9df-966ac5a0a93b"}
","requestId":"025fe01c-a01e-423e-b9df-966ac5a0a93b","timestamp":"2024-02-02T22:22:11Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=d756fe60-fe0b-4939-b757-0e2d911f8f3f] cannot determine cloud provider ","requestId":"d756fe60-fe0b-4939-b757-0e2d911f8f3f","timestamp":"2024-02-02T22:22:12Z"}[0m
2024-02-02 22:22:24 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=b253e0ba-7554-4a49-a119-dcf7c7f4860b] could not obtain default workload version from configuration variables in DB","requestId":"b253e0ba-7554-4a49-a119-dcf7c7f4860b","timestamp":"2024-02-02T22:22:12Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:4647","message":"[requestId=5aab45c9-a522-40e1-b953-0b58e9d0c9d8] cannot determine cloud provider ","requestId":"5aab45c9-a522-40e1-b953-0b58e9d0c9d8","timestamp":"2024-02-02T22:22:12Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:187","message":"[requestId=f27fee86-37a0-4304-8e83-c59e1bdfde0a] failed to start RestoreWorkspace: Cannot restore workspace with endpoint endpoint1, as the endpoint is currently being used by workspace test-workspace. Delete or suspend test-workspace to use the endpoint.
","requestId":"f27fee86-37a0-4304-8e83-c59e1bdfde0a","timestamp":"2024-02-02T22:22:12Z"}[0m
2024-02-02 22:22:24 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=a5255cc2-b845-4b3a-b5b5-73bcaf9aa176] could not obtain default workload version from configuration variables in DB","requestId":"a5255cc2-b845-4b3a-b5b5-73bcaf9aa176","timestamp":"2024-02-02T22:22:12Z"}[0m
2024-02-02 22:22:24 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=d77de362-f543-449d-b513-946e5645eb5b] could not obtain default workload version from configuration variables in DB","requestId":"d77de362-f543-449d-b513-946e5645eb5b","timestamp":"2024-02-02T22:22:12Z"}[0m
2024-02-02 22:22:24 - INFO-root::util|431:: 2024/02/02 22:22:12 Failed to start Create: Cannot create workspace with endpoint endpoint1, as the endpoint is currently being used by workspace test workspace. Delete or suspend test workspace to use the endpoint.[0m
2024-02-02 22:22:24 - INFO-root::util|431:: 2024/02/02 22:22:12 Failed to start Create: Cannot create workspace with endpoint endpoint1, as the endpoint is currently being used by workspace test workspace. Delete or suspend test workspace to use the endpoint.[0m
2024-02-02 22:22:24 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:160","message":"[requestId=f68be9ce-0306-42a6-b4cf-44f7d4842194] Failed to start Resume: Cannot resume workspace with endpoint endpointresume, as the endpoint is currently being used by workspace test workspace. Delete or suspend test workspace to use the endpoint.
","requestId":"f68be9ce-0306-42a6-b4cf-44f7d4842194","timestamp":"2024-02-02T22:22:12Z"}[0m
2024-02-02 22:22:24 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:user:1f77129b-9cb0-4a23-970f-a5f8af88fdea","level":"warning","line":"dynamicaddon/dynamicaddon.go:176","message":"[requestId=fake-request-id] Failed to lookup Spark addons: Get "/api/v1/latestComponentInfo": unsupported protocol scheme ""","requestId":"fake-request-id","timestamp":"2024-02-02T22:22:13Z"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestRotateCredentialsWorkflow/Error_in_update_credentials_activity[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestRotateCredentialsWorkflow/Error_in_restart_deployment_activity[0m
2024-02-02 22:22:25 - INFO-root::util|431::     --- PASS: TestRotateCredentialsWorkflow/Error_in_update_credentials_activity (0.01s)[0m
2024-02-02 22:22:25 - INFO-root::util|431::     --- PASS: TestRotateCredentialsWorkflow/Error_in_restart_deployment_activity (0.01s)[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestUpdateNodeImageWorkflow/error_initiating_node_refresh[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestUpdateNodeImageWorkflow/Cluster_landed_in_UPDATE_FAILED_status[0m
2024-02-02 22:22:25 - INFO-root::util|431:: {"level":"info","ts":1706912533.3243172,"caller":"core/update_node_image_workflow.go:121","msg":"cluster status: {Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status:UPDATING Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}","requestId":"request-id","actorCrn":"actor-crn","resourceId":"cool-workspace-name"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestUpdateNodeImageWorkflow/Cluster_nodes_successfully_rolled/RefreshLoadBalancerActivity_failed[0m
2024-02-02 22:22:25 - INFO-root::util|431:: {"level":"info","ts":1706912533.335688,"caller":"core/update_node_image_workflow.go:121","msg":"cluster status: {Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status:UPDATING Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}","requestId":"request-id","actorCrn":"actor-crn","resourceId":"cool-workspace-name"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestUpdateNodeImageWorkflow/Cluster_nodes_successfully_rolled/CreateFreeIpaDnsEntryActivity_failed[0m
2024-02-02 22:22:25 - INFO-root::util|431:: {"level":"info","ts":1706912533.343037,"caller":"core/update_node_image_workflow.go:121","msg":"cluster status: {Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status:UPDATING Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}","requestId":"request-id","actorCrn":"actor-crn","resourceId":"cool-workspace-name"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: {"level":"info","ts":1706912533.3497796,"caller":"core/update_node_image_workflow.go:121","msg":"cluster status: {Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status:UPDATING Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}","requestId":"request-id","actorCrn":"actor-crn","resourceId":"cool-workspace-name"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: {"level":"info","ts":1706912533.3559995,"caller":"core/update_node_image_workflow.go:121","msg":"cluster status: {Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status:UPDATING Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}","requestId":"request-id","actorCrn":"actor-crn","resourceId":"cool-workspace-name"}[0m
2024-02-02 22:22:25 - INFO-root::util|431::     --- PASS: TestUpdateNodeImageWorkflow/error_initiating_node_refresh (0.01s)[0m
2024-02-02 22:22:25 - INFO-root::util|431::     --- PASS: TestUpdateNodeImageWorkflow/Cluster_landed_in_UPDATE_FAILED_status (0.01s)[0m
2024-02-02 22:22:25 - INFO-root::util|431::         --- PASS: TestUpdateNodeImageWorkflow/Cluster_nodes_successfully_rolled/RefreshLoadBalancerActivity_failed (0.01s)[0m
2024-02-02 22:22:25 - INFO-root::util|431::         --- PASS: TestUpdateNodeImageWorkflow/Cluster_nodes_successfully_rolled/CreateFreeIpaDnsEntryActivity_failed (0.01s)[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.360Z	INFO	core/workspace_activities.go:127	Fetched cluster status information: &{Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[liftieinfra:{InstanceGroupName:liftieinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlcpu:{InstanceGroupName:mlcpu InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlinfra:{InstanceGroupName:mlinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]}] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status: Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.363Z	ERROR	core/workspace_activities.go:156	Failed to convert byte data of workspace auto-scaling info to JSON data: unexpected end of JSON input	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.369Z	ERROR	core/workspace_activities.go:280	Failed to convert byte data of workspace replica info to JSON: unexpected end of JSON input	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestWorkspaceActivities/Test_Activity_InitiateNodeRefreshActivity/error_in_getting_cluster_status[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.377Z	INFO	core/workspace_activities.go:514	Fetched cluster status: &{Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[liftieinfra:{InstanceGroupName:liftieinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlcpu:{InstanceGroupName:mlcpu InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:0 MinInstances:0 MaxInstances:1 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlinfra:{InstanceGroupName:mlinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:0 MinInstances:0 MaxInstances:1 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]}] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status: Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.378Z	INFO	core/workspace_activities.go:514	Fetched cluster status: &{Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[liftieinfra:{InstanceGroupName:liftieinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlcpu:{InstanceGroupName:mlcpu InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlinfra:{InstanceGroupName:mlinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]}] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status: Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - [31mERROR-root::util|431:: 2024-02-02T22:22:13.380Z	ERROR	core/workspace_activities.go:614	Encountered error while deleting FreeIpa DNS entry: unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.381Z	INFO	core/workspace_activities.go:651	Fetched cluster status: &{Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[liftieinfra:{InstanceGroupName:liftieinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlcpu:{InstanceGroupName:mlcpu InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlinfra:{InstanceGroupName:mlinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]}] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status: Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.382Z	INFO	core/workspace_activities.go:651	Fetched cluster status: &{Cloud:{SecurityGroups:[] WorkerSecurityGroup: WorkerSubnets:[] Subnets:[] DisplayName: InstanceGroups:map[liftieinfra:{InstanceGroupName:liftieinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:2 MinInstances:2 MaxInstances:3 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlcpu:{InstanceGroupName:mlcpu InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:0 MinInstances:0 MaxInstances:1 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]} mlinfra:{InstanceGroupName:mlinfra InstanceType: InstanceTypes:[] InstanceTier: InstanceCount:0 MinInstances:0 MaxInstances:1 Labels:map[] Instances:[] Tags:[] Pools:[] NodePool:{Name: Type:} NodeVolumeSize:0 Taints:map[]}] Tags:map[] ResourceGroup: Region: FipsModeEnabled:false} ClusterStatus:{Status: Message:} FailedEvents:[] Kubernetes:{Version:} Upgrade:{Available:[]} ImagePullSecrets:[] ClusterStateVersion:0}	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestWorkspaceActivities/Test_Activity_CreateFreeIpaDnsEntry/When_FreeIPA_DNS_entry_creataion_throws_error[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.385Z	ERROR	core/workspace_activities.go:841	Failed to create DNS entry in FreeIpa: unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.389Z	ERROR	core/workspace_activities.go:998	Instance group of name mlgpu is not present in the cluster.	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.392Z	ERROR	core/workspace_activities.go:1373	Unable to fetch block volume info of for workspaceID '%!s(int64=1)' in namespace mlx due to error : unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.393Z	ERROR	core/workspace_activities.go:1407	Unable to fetch persistent volumes for workspaceID '%!s(int64=1)' in namespace mlx due to error : unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.396Z	ERROR	core/workspace_activities.go:1442	Unable to fetch persistent volume claims for workspaceID '%!s(int64=1)' in namespace mlx due to error : unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.397Z	ERROR	core/workspace_activities.go:370	Failed to fetch the pods from namespace mlx: unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.400Z	ERROR	core/workspace_activities.go:422	Failed to fetch the jobs from namespace mlx: unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.403Z	ERROR	core/workspace_activities.go:1500	Unable to fetch secrets for workspaceID '%!s(int64=1)' in namespace mlx due to error : unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.412Z	ERROR	core/workspace_activities.go:1545	unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.413Z	ERROR	core/workspace_activities.go:1546	Unable to patch k8s secret: internal-secrets	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.416Z	ERROR	core/workspace_activities.go:1580	unexpected error	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.416Z	ERROR	core/workspace_activities.go:1581	Error restarting the deployment: web	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431:: === RUN   TestWorkspaceActivities/Test_Activity_WaitForDeploymentByNameActivity/Failure[0m
2024-02-02 22:22:25 - INFO-root::util|431:: 2024-02-02T22:22:13.418Z	ERROR	core/workspace_activities.go:1618	The deployment failed to restart	{"LocalActivityID": "test-local-activity", "LocalActivityType": "", "WorkflowType": "workflow-type-not-specified", "WorkflowID": "default-test-workflow-id", "RunID": "default-test-run-id", "requestId": "56beb732-05d5-4b0a-8e44-88905c2bfd0e", "actorCrn": "crn:altus:iam:us-west-1:clouderaTestTenant:user:xxxTentacion", "resourceId": "crn:cdp:ml:us-west-1:xxxTentacion:workspace:8d6b1854-8e2f-44ba-9fc2-4b5f502784ea"}[0m
2024-02-02 22:22:25 - INFO-root::util|431::         --- PASS: TestWorkspaceActivities/Test_Activity_InitiateNodeRefreshActivity/error_in_getting_cluster_status (0.00s)[0m
2024-02-02 22:22:25 - INFO-root::util|431::         --- PASS: TestWorkspaceActivities/Test_Activity_CreateFreeIpaDnsEntry/When_FreeIPA_DNS_entry_creataion_throws_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::         --- PASS: TestWorkspaceActivities/Test_Activity_WaitForDeploymentByNameActivity/Failure (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieClusterProvisionerActivities_GetLiftieIDForWorkspace/should_return_error_when_liftieID_does_not_exist[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"level":"error","ts":1706912467.3768034,"caller":"cluster/provision_cluster_activities.go:51","msg":"No associated cluster found for workspaceId '456', workspaceCrn '%!s(MISSING)'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.(*LiftieClusterProvisionerActivities).GetLiftieIDForWorkspace
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities.go:51
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.TestLiftieClusterProvisionerActivities_GetLiftieIDForWorkspace.func7
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities_test.go:102
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieClusterProvisionerActivities_GetLiftieIDForWorkspace/should_return_error_when_an_unexpected_error_occurs[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"level":"error","ts":1706912467.3787215,"caller":"cluster/provision_cluster_activities.go:51","msg":"No associated cluster found for workspaceId '789', workspaceCrn '%!s(MISSING)'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.(*LiftieClusterProvisionerActivities).GetLiftieIDForWorkspace
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities.go:51
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.TestLiftieClusterProvisionerActivities_GetLiftieIDForWorkspace.func7
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities_test.go:102
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieClusterProvisionerActivities_GetLiftieIDForWorkspace/should_return_error_when_liftieID_does_not_exist (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieClusterProvisionerActivities_GetLiftieIDForWorkspace/should_return_error_when_an_unexpected_error_occurs (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieClusterProvisionerActivities_DoesClusterExistActivity/should_return_error_when_GetCluster_throws_a_non-404_error[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"level":"error","ts":1706912467.3829193,"caller":"cluster/provision_cluster_activities.go:70","msg":"Failed to get cluster details from Liftie: error not-404","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.(*LiftieClusterProvisionerActivities).DoesClusterExistActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities.go:70
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.TestLiftieClusterProvisionerActivities_DoesClusterExistActivity.func7
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities_test.go:193
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieClusterProvisionerActivities_DoesClusterExistActivity/should_return_error_when_GetCluster_throws_a_non-404_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieClusterProvisionerActivities_GetVersion/should_throw_error[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieClusterProvisionerActivities_GetVersion/should_throw_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieClusterProvisionerActivities_FetchClusterStatusActivity/should_return_error_when_GetCluster_throws_an_error[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"level":"error","ts":1706912467.3870642,"caller":"cluster/provision_cluster_activities.go:117","msg":"failed to get cluster details from Liftie for clusterId '': error not-404","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.(*LiftieClusterProvisionerActivities).FetchClusterStatusActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities.go:117
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.TestLiftieClusterProvisionerActivities_FetchClusterStatusActivity.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities_test.go:325
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieClusterProvisionerActivities_FetchClusterStatusActivity/should_return_error_when_GetCluster_throws_an_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieClusterProvisionerActivities_CreateClusterActivity/should_throw_error_when_ProvisionAsync_fails[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"level":"error","ts":1706912467.3911152,"caller":"cluster/provision_cluster_activities.go:83","msg":"Creation of cluster failed due to error 'ProvisionAsync error'","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.(*LiftieClusterProvisionerActivities).CreateClusterActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities.go:83
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.TestLiftieClusterProvisionerActivities_CreateClusterActivity.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities_test.go:407
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieClusterProvisionerActivities_CreateClusterActivity/should_throw_error_when_ProvisionAsync_fails (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieClusterProvisionerActivities_UpdateClusterIdForWorkspaceIdActivity/should_throw_error[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"level":"error","ts":1706912467.3949723,"caller":"cluster/provision_cluster_activities.go:129","msg":"Failed to set liftieId to '' for workspaceId '0', workspaceCrn '': SetLitieId error","requestId":"request-id","actorCrn":"actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.LiftieClusterProvisionerActivities.UpdateClusterIdForWorkspaceIdActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities.go:129
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster.TestLiftieClusterProvisionerActivities_UpdateClusterIdForWorkspaceIdActivity.func5
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/core/cluster/provision_cluster_activities_test.go:478
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieClusterProvisionerActivities_UpdateClusterIdForWorkspaceIdActivity/should_throw_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestProvisionClusterWorkflow/TestLiftieClusterProvisionerWorkflow_ProvisionClusterForResourceFails_AfterClusterStatusFoundInFailedState[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: 2024-02-02T22:21:07.417Z	ERROR	cluster/provision_cluster_workflow.go:147	Encountered error when checking for cluster readiness for clusterId 'liftie-1234' for resourceCrn 'resourceCrn': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestProvisionClusterWorkflow/TestLiftieClusterProvisionerWorkflow_ProvisionClusterForResourceFails_ClusterAssociatedWithResourceButLiftieThrowsError[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.422Z	ERROR	cluster/provision_cluster_workflow.go:99	Cluster 'liftie-1234' is associated with resource 'resourceCrn', but not found in downstream service Liftie/Computex	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.427Z	ERROR	cluster/provision_cluster_workflow.go:99	Cluster 'liftie-1234' is associated with resource 'resourceCrn', but not found in downstream service Liftie/Computex	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.432Z	ERROR	cluster/provision_cluster_workflow.go:113	Failed to create cluster for resourceCrn 'resourceCrn': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.442Z	ERROR	cluster/provision_cluster_workflow.go:175	cluster 'liftie-1234' for resourceCrn 'resourceCrn' failed to become ready after '1500' seconds	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.447Z	ERROR	cluster/provision_cluster_workflow.go:128	Failed to update clusterId to 'liftie-1234' for resourceCrn 'resourceCrn': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: 2024-02-02T22:21:07.457Z	ERROR	cluster/provision_cluster_workflow.go:147	Encountered error when checking for cluster readiness for clusterId 'liftie-1234' for resourceCrn 'resourceCrn': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: 2024-02-02T22:21:07.465Z	ERROR	cluster/provision_cluster_workflow.go:147	Encountered error when checking for cluster readiness for clusterId 'liftie-1234' for resourceCrn 'resourceCrn': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestProvisionClusterWorkflow/TestLiftieClusterProvisionerWorkflow_ProvisionClusterForWorkspaceFails_AfterClusterStatusFoundInFailedState[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: 2024-02-02T22:21:07.474Z	ERROR	cluster/provision_cluster_workflow.go:293	Encountered error when checking for cluster readiness for clusterId 'liftie-1234' for workspaceId '1', workspaceCrn '': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestProvisionClusterWorkflow/TestLiftieClusterProvisionerWorkflow_ProvisionClusterForWorkspaceFails_ClusterAssociatedWithWorkspaceButLiftieThrowsError[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.479Z	ERROR	cluster/provision_cluster_workflow.go:245	Cluster 'liftie-1234' is associated with workspaceId '1', workspaceCrn 'workspace-crn', but not found in downstream service Liftie/Computex	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.484Z	ERROR	cluster/provision_cluster_workflow.go:245	Cluster 'liftie-1234' is associated with workspaceId '1', workspaceCrn 'workspace-crn', but not found in downstream service Liftie/Computex	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.491Z	ERROR	cluster/provision_cluster_workflow.go:259	Failed to create cluster for workspaceId '1', workspaceCrn 'workspace-crn': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.501Z	ERROR	cluster/provision_cluster_workflow.go:326	cluster 'liftie-1234' for workspaceId '1', workspaceCrn 'workspace-crn' failed to become ready after '1500' seconds	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: 2024-02-02T22:21:07.511Z	ERROR	cluster/provision_cluster_workflow.go:274	Failed to update clusterId to 'liftie-1234' for workspaceId '1', workspaceCrn 'workspace-crn': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: 2024-02-02T22:21:07.522Z	ERROR	cluster/provision_cluster_workflow.go:293	Encountered error when checking for cluster readiness for clusterId 'liftie-1234' for workspaceId '1', workspaceCrn '': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: 2024-02-02T22:21:07.528Z	ERROR	cluster/provision_cluster_workflow.go:293	Encountered error when checking for cluster readiness for clusterId 'liftie-1234' for workspaceId '1', workspaceCrn '': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-workspace-name"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestProvisionClusterWorkflow/TestLiftieClusterProvisionerWorkflow_ProvisionClusterForResourceFails_AfterClusterStatusFoundInFailedState (0.01s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestProvisionClusterWorkflow/TestLiftieClusterProvisionerWorkflow_ProvisionClusterForResourceFails_ClusterAssociatedWithResourceButLiftieThrowsError (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestProvisionClusterWorkflow/TestLiftieClusterProvisionerWorkflow_ProvisionClusterForWorkspaceFails_AfterClusterStatusFoundInFailedState (0.01s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestProvisionClusterWorkflow/TestLiftieClusterProvisionerWorkflow_ProvisionClusterForWorkspaceFails_ClusterAssociatedWithWorkspaceButLiftieThrowsError (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieDeployments_GetAllDeploymentsLegacy/Test_error_from_http[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieDeployments_GetAllDeploymentsLegacy/Test_response_unmarshall_error[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieDeployments_GetAllDeploymentsLegacy/Test_error_from_http (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieDeployments_GetAllDeploymentsLegacy/Test_response_unmarshall_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieDeployments_PutDeploymentLegacy/Test_error_from_http[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:251","message":"[requestId=unit-test-Test error from http] Deployment '' will be upgraded to version ''","requestId":"unit-test-Test error from http","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:265","message":"[requestId=unit-test-Test error from http] Update request '{ false false   map[] 0 0  false}'","requestId":"unit-test-Test error from http","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieDeployments_PutDeploymentLegacy/Test_response_unmarshall_error[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:251","message":"[requestId=unit-test-Test response unmarshall error] Deployment '' will be upgraded to version ''","requestId":"unit-test-Test response unmarshall error","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:265","message":"[requestId=unit-test-Test response unmarshall error] Update request '{ false false   map[] 0 0  false}'","requestId":"unit-test-Test response unmarshall error","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieDeployments_PutDeploymentLegacy/Test_error_from_http (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieDeployments_PutDeploymentLegacy/Test_response_unmarshall_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_GetLatestChartVersionsLegacy/Test_error[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_GetLatestChartVersionsLegacy/More_than_1_available_upgrades_throws_error[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_GetLatestChartVersionsLegacy/Test_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_GetLatestChartVersionsLegacy/More_than_1_available_upgrades_throws_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_UpdateChartsLegacy/PutDeployment_throws_error[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_UpdateChartsLegacy/PutDeployment_throws_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_AreChartsUpdatedLegacy/Test_1_managed_chart_failed_deployment[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:172","message":"[requestId=unit-test-Test 1 managed chart failed deployment] DeploymentListResponse '{liftie-v00rlylj [{ 0.0.3-dsp.3 webhook for proxy ConfigMaps  cdp-proxy-webhook map[]}] [{{ 1.0.0-b9343-dsp.1 workload client application to output metering heartbeat events periodically.  cdp-metering map[]} true  cdp-metering metering map[AWSLoadBalancerType:nlb] 4 FAILED []}] []}'","requestId":"unit-test-Test 1 managed chart failed deployment","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=unit-test-Test 1 managed chart failed deployment] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"unit-test-Test 1 managed chart failed deployment","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:196","message":"[requestId=unit-test-Test 1 managed chart failed deployment] Deployment of chart 'cdp-metering' is in state 'FAILED'","requestId":"unit-test-Test 1 managed chart failed deployment","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:172","message":"[requestId=unit-test-Test 1 managed chart wrong version] DeploymentListResponse '{liftie-v00rlylj [{ 0.0.3-dsp.3 webhook for proxy ConfigMaps  cdp-proxy-webhook map[]}] [{{ 1.0.0-b9343-dsp.1 workload client application to output metering heartbeat events periodically.  cdp-metering map[]} true  cdp-metering metering map[AWSLoadBalancerType:nlb] 4 FAILED []}] []}'","requestId":"unit-test-Test 1 managed chart wrong version","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=unit-test-Test 1 managed chart wrong version] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"unit-test-Test 1 managed chart wrong version","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:196","message":"[requestId=unit-test-Test 1 managed chart wrong version] Deployment of chart 'cdp-metering' is in state 'FAILED'","requestId":"unit-test-Test 1 managed chart wrong version","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:172","message":"[requestId=unit-test-Test 1 non-managed chart] DeploymentListResponse '{liftie-v00rlylj [{ 0.0.3-dsp.3 webhook for proxy ConfigMaps  cdp-proxy-webhook map[]}] [{{ 1.0.0-b9343-dsp.1 workload client application to output metering heartbeat events periodically.  cdp-metering map[]} false  cdp-metering metering map[AWSLoadBalancerType:nlb] 4 FAILED []}] []}'","requestId":"unit-test-Test 1 non-managed chart","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_AreChartsUpdatedLegacy/Test_GetAllDeployments_errors[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=unit-test-Test 1 managed chart UNKNOWN status] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"unit-test-Test 1 managed chart UNKNOWN status","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=unit-test-Test 1 managed chart UNINSTALLED status] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"unit-test-Test 1 managed chart UNINSTALLED status","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=unit-test-Test 1 managed chart SUPERSEDED status] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"unit-test-Test 1 managed chart SUPERSEDED status","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=unit-test-Test 1 managed chart PENDING-INSTALL status] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"unit-test-Test 1 managed chart PENDING-INSTALL status","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=unit-test-Test 1 managed chart PENDING-ROLLBACK status] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"unit-test-Test 1 managed chart PENDING-ROLLBACK status","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_AreChartsUpdatedLegacy/Test_1_managed_chart_failed_deployment (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_AreChartsUpdatedLegacy/Test_GetAllDeployments_errors (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterWorkflow_UpdateClusterLegacy/Test_GetChartToNamespaceMapForUpgradeLegacy_error[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterWorkflow_UpdateClusterLegacy/Test_UpdateChartsToLatestVersionLegacy_error[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterWorkflow_UpdateClusterLegacy/Test_AreChartsUpdatedToLatestVersionLegacy_error[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"cluster/update_cluster_activities.go:71","message":"[requestId=unit-test-Test AreChartsUpdatedToLatestVersionLegacy error] Checking for deployment status. Attempt '1/1'","requestId":"unit-test-Test AreChartsUpdatedToLatestVersionLegacy error","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:75","message":"[requestId=unit-test-Test AreChartsUpdatedToLatestVersionLegacy error] Some deployments failed to update successfully. Error 'AreChartsUpdatedToLatestVersionLegacy error'","requestId":"unit-test-Test AreChartsUpdatedToLatestVersionLegacy error","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:75","message":"[requestId=unit-test-Test success after 5 attempts] Some deployments failed to update successfully. Error 'not ready'","requestId":"unit-test-Test success after 5 attempts","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:75","message":"[requestId=unit-test-Test success after 5 attempts] Some deployments failed to update successfully. Error 'not ready'","requestId":"unit-test-Test success after 5 attempts","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:75","message":"[requestId=unit-test-Test success after 5 attempts] Some deployments failed to update successfully. Error 'not ready'","requestId":"unit-test-Test success after 5 attempts","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"cluster/update_cluster_activities.go:75","message":"[requestId=unit-test-Test success after 5 attempts] Some deployments failed to update successfully. Error 'not ready'","requestId":"unit-test-Test success after 5 attempts","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterWorkflow_UpdateClusterLegacy/Test_GetChartToNamespaceMapForUpgradeLegacy_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterWorkflow_UpdateClusterLegacy/Test_UpdateChartsToLatestVersionLegacy_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterWorkflow_UpdateClusterLegacy/Test_AreChartsUpdatedToLatestVersionLegacy_error (0.00s)[0m
2024-02-02 22:22:26 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_AreChartsUpdated/Test_1_managed_chart_failed_deployment[0m
2024-02-02 22:22:26 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"info","line":"cluster/update_cluster_activities.go:172","message":"[requestId=request-id] DeploymentListResponse '{liftie-v00rlylj [{ 0.0.3-dsp.3 webhook for proxy ConfigMaps  cdp-proxy-webhook map[]}] [{{ 1.0.0-b9343-dsp.1 workload client application to output metering heartbeat events periodically.  cdp-metering map[]} true  cdp-metering metering map[AWSLoadBalancerType:nlb] 4 FAILED []}] []}'","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=request-id] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"info","line":"cluster/update_cluster_activities.go:196","message":"[requestId=request-id] Deployment of chart 'cdp-metering' is in state 'FAILED'","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"info","line":"cluster/update_cluster_activities.go:172","message":"[requestId=request-id] DeploymentListResponse '{liftie-v00rlylj [{ 0.0.3-dsp.3 webhook for proxy ConfigMaps  cdp-proxy-webhook map[]}] [{{ 1.0.0-b9343-dsp.1 workload client application to output metering heartbeat events periodically.  cdp-metering map[]} true  cdp-metering metering map[AWSLoadBalancerType:nlb] 4 FAILED []}] []}'","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"actor-crn","level":"error","line":"cluster/update_cluster_activities.go:195","message":"[requestId=request-id] Deployment of chart 'cdp-metering' failed to update to version '1.0.0-b9343-dsp.1'","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"info","line":"cluster/update_cluster_activities.go:196","message":"[requestId=request-id] Deployment of chart 'cdp-metering' is in state 'FAILED'","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"info","line":"cluster/update_cluster_activities.go:172","message":"[requestId=request-id] DeploymentListResponse '{liftie-v00rlylj [{ 0.0.3-dsp.3 webhook for proxy ConfigMaps  cdp-proxy-webhook map[]}] [{{ 1.0.0-b9343-dsp.1 workload client application to output metering heartbeat events periodically.  cdp-metering map[]} false  cdp-metering metering map[AWSLoadBalancerType:nlb] 4 FAILED []}] []}'","requestId":"request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_AreChartsUpdated/Test_GetAllDeployments_errors[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_AreChartsUpdated/Test_1_managed_chart_failed_deployment (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_AreChartsUpdated/Test_GetAllDeployments_errors (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_GetLatestChartVersions/Test_error[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_GetLatestChartVersions/Test_error (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestLiftieUpdateClusterActivity_UpdateCharts/PutDeployment_throws_error[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestLiftieUpdateClusterActivity_UpdateCharts/PutDeployment_throws_error (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestLogClusterStatus/failures_with_empty_notreadypodlist[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cluster_logging/cluster_logging.go:69","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] Failed to get ingress status:  unexpected error","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cluster_logging/cluster_logging.go:117","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] Failed to retrieve ingress LB events: unexpected error","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cluster_logging/cluster_logging.go:81","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] Failed to get pod status:  unexpected error","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cluster_logging/cluster_logging.go:85","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] Unable to find not ready pods","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestLogClusterStatus/failures_with_non_empty_notreadypodlist[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cluster_logging/cluster_logging.go:69","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] Failed to get ingress status:  unexpected error","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cluster_logging/cluster_logging.go:117","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] Failed to retrieve ingress LB events: unexpected error","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cluster_logging/cluster_logging.go:187","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] Failed to get events for pod mlx/test_pod: unexpected error","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"error","line":"cluster_logging/cluster_logging.go:132","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] Failed to get container logs from mlx/test_pod: unexpected error","pod":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"test_pod","creationTimestamp":null},"spec":{"containers":[{"name":"test_container","resources":{}}]},"status":{"containerStatuses":[{"name":"","state":{},"lastState":{},"ready":true,"restartCount":8,"image":"","imageID":""}]}},"requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"info","line":"cluster_logging/cluster_logging.go:108","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] {"IngressError":{"IngressName":"","IngressStatus":"","IngressEvents":[""]},"NotReadyPods":["test_pod"],"PodError":{"test_pod":{"PodStatus":{"Name":"test_pod","Ready":"1/1","Status":"","Restarts":8,"Age":"2562047h47m16.854775807s"},"PodEvents":null,"ContainerLogs":{}}}}","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","level":"info","line":"cluster_logging/cluster_logging.go:108","message":"[requestId=1dfdf1c1-b15d-458b-b7ef-3727a6323e12] {"IngressError":{"IngressName":"ingress-traefik","IngressStatus":"\u0026ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{LoadBalancerIngress{IP:0.0.0.0,Hostname:,Ports:[]PortStatus{},},},},Conditions:[]Condition{},}","IngressEvents":["event1","event2"]},"NotReadyPods":["test_pod"],"PodError":{"test_pod":{"PodStatus":{"Name":"test_pod","Ready":"1/1","Status":"","Restarts":8,"Age":"2562047h47m16.854775807s"},"PodEvents":["Normal\t1970-01-01 00:00:00 +0000 UTC\ttest event\tThis is a test event"],"ContainerLogs":{}}}}","requestId":"1dfdf1c1-b15d-458b-b7ef-3727a6323e12","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestLogClusterStatus/failures_with_empty_notreadypodlist (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestLogClusterStatus/failures_with_non_empty_notreadypodlist (0.01s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: 2024/02/02 22:21:07 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:2884 [35;1mUNIQUE constraint failed: mlx_configuration.variable_name[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"warning","line":"mutators/environmentlist_mutator.go:163","message":"[requestId=req-id] Failed to get Chart Version for version gbn://34567. Getting following Error : random error during fetching build version from GBN","requestId":"req-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"actor-crn","level":"warning","line":"mutators/workspace_mutator.go:111","message":"[requestId=req-id] Failed to get Chart Version for version gbn://34567. Getting following Error : random error during fetching build version from GBN","requestId":"req-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestSnapshotResourceGroupValidator/error_during_checking_for_resource_group[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === PAUSE TestSnapshotResourceGroupValidator/error_during_checking_for_resource_group[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === CONT  TestSnapshotResourceGroupValidator/error_during_checking_for_resource_group[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestSnapshotResourceGroupValidator/error_during_checking_for_resource_group (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestDeleteBackupUserRoleValidator/UMS_failure[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestDeleteBackupUserRoleValidator/validation_failure[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"validator/validator.go:1778","message":"[requestId=a24ebdc8-5b2a-48ac-a2de-ac49b4a7d01f] User role validation result: 'FAILED', message: 'User role validations failed.'","requestId":"a24ebdc8-5b2a-48ac-a2de-ac49b4a7d01f","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestDeleteBackupUserRoleValidator/UMS_failure (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestDeleteBackupUserRoleValidator/validation_failure (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestBackupSnapshotsPresentValidator/validation_failed[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"validator/validator.go:1850","message":"[requestId=b4255b09-4fab-475c-9352-bfb331e79af2] backup snapshot validation error: random error","requestId":"b4255b09-4fab-475c-9352-bfb331e79af2","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestBackupSnapshotsPresentValidator/validation_failed (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestCreateBackupIAMPermissionValidator/validation_failed[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestCreateBackupIAMPermissionValidator/validation_failed (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestDeleteBackupIAMPermissionValidator/validation_failed[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestDeleteBackupIAMPermissionValidator/validation_failed (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestRestoreWorkspaceIAMPermissionValidator/validation_failed[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestRestoreWorkspaceIAMPermissionValidator/validation_failed (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestUserRoleValidator/UMS_failure[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestUserRoleValidator/validation_failure[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"warning","line":"validator/validator.go:611","message":"[requestId=ff558a57-d3bd-4283-b2f1-cfa67ac48703] User role validation result: 'FAILED', message: 'User role validations failed.'","requestId":"ff558a57-d3bd-4283-b2f1-cfa67ac48703","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestUserRoleValidator/UMS_failure (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestUserRoleValidator/validation_failure (0.00s)[0m
2024-02-02 22:22:27 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1936","message":"[requestId=18442530-6d7b-4781-ab7f-daafefd4b6bc] liftie pre-flight validation response: {"result":"PASSED","summary":{"passed":2,"warning":0,"failed":0,"skipped":1,"total":3},"message":"The cluster validation has passed, but some checks were skipped","validations":[{"name":"CSI IAM Policy Permissions Check","description":"For EKS \u003e= 1.23, when the entitlement LIFTIE_USE_PRECREATED_IAM_RESOURCES is enabled, the instance profile role cdp-liftie-instance-profile should have the necessary CSI polices attached.","category":"IAM","status":"SKIPPED","message":"CSI IAM Policy validation skipped.","detailedMessage":"CSI IAM Policy validation skipped. The entitlement 'LIFTIE_USE_PRECREATED_IAM_RESOURCES' is disabled. Skipping CSI IAM policy permissions check.","duration":"3µs"},{"name":"Kubernetes Version","description":"Each cloud provider (Amazon, Azure, Google, etc) supports different versions of Kubernetes.","category":"COMMON","status":"PASSED","message":"Amazon supports Kubernetes 1.24","detailedMessage":"Amazon supports Kubernetes 1.24","duration":"23µs"},{"name":"IAM Policy Permissions Check","description":"For a given role, the attached policies should have all the necessary permissions of a default cross account role in AWS ","category":"IAM","status":"PASSED","message":"IAM Policy validated.","detailedMessage":"IAM Resource Policy validation completed, verified if the corresponding profile exists and has the necessary permissions.","duration":"1.91s"}]}","requestId":"18442530-6d7b-4781-ab7f-daafefd4b6bc","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestValidateLiftieUpgrade/liftie_preflight_failure[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1936","message":"[requestId=8ef95574-dba0-49d7-96eb-8736f262973f] liftie pre-flight validation response: {"result":"FAILED","summary":{"passed":1,"warning":0,"failed":1,"skipped":1,"total":3},"message":"The cluster validation has failed. One or more checks, which are required to pass in order to ensure that a cluster is created successfully, has failed.","validations":[{"name":"CSI IAM Policy Permissions Check","description":"For EKS \u003e= 1.23, when the entitlement LIFTIE_USE_PRECREATED_IAM_RESOURCES is enabled, the instance profile role cdp-liftie-instance-profile should have the necessary CSI polices attached.","category":"IAM","status":"SKIPPED","message":"CSI IAM Policy validation skipped.","detailedMessage":"CSI IAM Policy validation skipped. The entitlement 'LIFTIE_USE_PRECREATED_IAM_RESOURCES' is disabled. Skipping CSI IAM policy permissions check.","duration":"3µs"},{"name":"Kubernetes Version","description":"Each cloud provider (Amazon, Azure, Google, etc) supports different versions of Kubernetes.","category":"COMMON","status":"PASSED","message":"Amazon supports Kubernetes 1.24","detailedMessage":"Amazon supports Kubernetes 1.24","duration":"23µs"},{"name":"IAM Policy Permissions Check","description":"For a given role, the attached policies should have all the necessary permissions of a default cross account role in AWS ","category":"IAM","status":"FAILED","message":"IAM Policy validated.","detailedMessage":"IAM Resource Policy validation completed, verified if the corresponding profile exists and has the necessary permissions.","duration":"1.91s"}]}","requestId":"8ef95574-dba0-49d7-96eb-8736f262973f","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1959","message":"[requestId=8ef95574-dba0-49d7-96eb-8736f262973f] Completed pre-flight validation request to liftie. Result FAILED","requestId":"8ef95574-dba0-49d7-96eb-8736f262973f","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestValidateLiftieUpgrade/liftie_preflight_failure (0.00s)[0m
2024-02-02 22:22:27 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1936","message":"[requestId=70c64910-04d7-4b7e-9023-c82720b43a9a] liftie pre-flight validation response: {"result":"PASSED","summary":{"passed":2,"warning":0,"failed":0,"skipped":1,"total":3},"message":"The cluster validation has passed, but some checks were skipped"}","requestId":"70c64910-04d7-4b7e-9023-c82720b43a9a","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestValidateLiftieUpdateSecurity/liftie_preflight_failure[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1936","message":"[requestId=d81d20fd-6a61-47f9-aa50-b8c4b56d5c7f] liftie pre-flight validation response: {"result":"FAILED","summary":{"passed":1,"warning":0,"failed":1,"skipped":1,"total":3},"message":"The cluster validation has failed. One or more checks, which are required to pass in order to ensure that a cluster is created successfully, has failed."}","requestId":"d81d20fd-6a61-47f9-aa50-b8c4b56d5c7f","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1959","message":"[requestId=d81d20fd-6a61-47f9-aa50-b8c4b56d5c7f] Completed pre-flight validation request to liftie. Result FAILED","requestId":"d81d20fd-6a61-47f9-aa50-b8c4b56d5c7f","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestValidateLiftieUpdateSecurity/liftie_preflight_failure (0.00s)[0m
2024-02-02 22:22:27 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1936","message":"[requestId=a8ea0963-74c3-4e68-a14c-8c71d36e7266] liftie pre-flight validation response: {"result":"PASSED","summary":{"passed":2,"warning":0,"failed":0,"skipped":1,"total":3},"message":"The cluster validation has passed, but some checks were skipped"}","requestId":"a8ea0963-74c3-4e68-a14c-8c71d36e7266","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: === RUN   TestValidateLiftieUpdateIntanceGroup/liftie_preflight_failure[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1936","message":"[requestId=67c9ea05-bf63-4da8-b572-59459d44a589] liftie pre-flight validation response: {"result":"FAILED","summary":{"passed":1,"warning":0,"failed":1,"skipped":1,"total":3},"message":"The cluster validation has failed. One or more checks, which are required to pass in order to ensure that a cluster is created successfully, has failed."}","requestId":"67c9ea05-bf63-4da8-b572-59459d44a589","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"validator/validator.go:1959","message":"[requestId=67c9ea05-bf63-4da8-b572-59459d44a589] Completed pre-flight validation request to liftie. Result FAILED","requestId":"67c9ea05-bf63-4da8-b572-59459d44a589","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431::     --- PASS: TestValidateLiftieUpdateIntanceGroup/liftie_preflight_failure (0.00s)[0m
2024-02-02 22:22:27 - INFO-root::util|431:: 2024/02/02 22:21:07 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:2884 [35;1mUNIQUE constraint failed: mlx_configuration.variable_name[0m
2024-02-02 22:22:27 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=7972d216-be9f-45e7-b24d-ae336aae72f4] could not obtain default workload version from configuration variables in DB","requestId":"7972d216-be9f-45e7-b24d-ae336aae72f4","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:27 - INFO-root::util|431:: 2024/02/02 22:21:08 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:27 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=7972d216-be9f-45e7-b24d-ae336aae72f4] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"7972d216-be9f-45e7-b24d-ae336aae72f4","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:08 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:28 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=7972d216-be9f-45e7-b24d-ae336aae72f4] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"7972d216-be9f-45e7-b24d-ae336aae72f4","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:08 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:28 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=0101842c-7f3b-42df-8cf4-a34f3d4e488f] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"0101842c-7f3b-42df-8cf4-a34f3d4e488f","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:28 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5241","message":"[requestId=0101842c-7f3b-42df-8cf4-a34f3d4e488f] Storage unmount error handler is called","requestId":"0101842c-7f3b-42df-8cf4-a34f3d4e488f","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5242","message":"[requestId=0101842c-7f3b-42df-8cf4-a34f3d4e488f] Storage unmounting failed: Broken provisioner does nothing
","requestId":"0101842c-7f3b-42df-8cf4-a34f3d4e488f","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:28 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5297","message":"[requestId=0101842c-7f3b-42df-8cf4-a34f3d4e488f] Storage deprovision error handler is called","requestId":"0101842c-7f3b-42df-8cf4-a34f3d4e488f","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:5298","message":"[requestId=0101842c-7f3b-42df-8cf4-a34f3d4e488f] Storage deprovision failed: Broken provisioner does nothing
","requestId":"0101842c-7f3b-42df-8cf4-a34f3d4e488f","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:08 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:28 - INFO-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=0101842c-7f3b-42df-8cf4-a34f3d4e488f] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"0101842c-7f3b-42df-8cf4-a34f3d4e488f","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestListWorkspaces/Check_Queries/Error_on_just_passing_IncludeDeletedWorkspaces[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestListWorkspaces/Check_Queries/Error_on_just_passing_IncludeDeletedWorkspaces (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestGetBackupsByWorkspaceCrn/db_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:09 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:2588 [35;1mrandom error[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestGetBackupsByWorkspaceCrn/db_error (0.01s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestHasAuditEvent/error[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestHasAuditEvent/error (0.01s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestGetBackupByBackupCrn/transaction_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:10 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:2547 [35;1mrandom error[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestGetBackupByBackupCrn/transaction_error (0.01s)[0m
2024-02-02 22:22:28 - [31mERROR-root::util|431:: 2024/02/02 22:21:10 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:2632 [35;1msome random error in DB[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestExecuteCallbackAndAppendWorkflowMetadata/CallbackFailureDoesNotAddNewMetadata[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestExecuteCallbackAndAppendWorkflowMetadata/TxBeginFailureDoesNotAddNewMetadata[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestExecuteCallbackAndAppendWorkflowMetadata/MetadataReadFailureDoesNotAddNewMetadata[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestExecuteCallbackAndAppendWorkflowMetadata/NewMetadataWriteFailureDoesNotAddNewMetadata[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestExecuteCallbackAndAppendWorkflowMetadata/TxCommitFailureDoesNotAddNewMetadata[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestExecuteCallbackAndAppendWorkflowMetadata/Non_terminal_existing_workflow_causes_an_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestExecuteCallbackAndAppendWorkflowMetadata/Error_when_describing_previous_workflow_causes_an_error[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestExecuteCallbackAndAppendWorkflowMetadata/CallbackFailureDoesNotAddNewMetadata (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestExecuteCallbackAndAppendWorkflowMetadata/TxBeginFailureDoesNotAddNewMetadata (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestExecuteCallbackAndAppendWorkflowMetadata/MetadataReadFailureDoesNotAddNewMetadata (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestExecuteCallbackAndAppendWorkflowMetadata/NewMetadataWriteFailureDoesNotAddNewMetadata (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestExecuteCallbackAndAppendWorkflowMetadata/TxCommitFailureDoesNotAddNewMetadata (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestExecuteCallbackAndAppendWorkflowMetadata/Non_terminal_existing_workflow_causes_an_error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestExecuteCallbackAndAppendWorkflowMetadata/Error_when_describing_previous_workflow_causes_an_error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestUpdateWorkflowMetadataStatus/no_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestUpdateWorkflowMetadataStatus/error_occurs[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:10 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:2819 [35;1mrandom error[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestUpdateWorkflowMetadataStatus/no_error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestUpdateWorkflowMetadataStatus/error_occurs (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestGetWorkflowMetadata/error_during_execution[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:17 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:2825 [35;1mrandom error[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestGetWorkflowMetadata/error_during_execution (0.01s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:20 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:2884 [35;1mUNIQUE constraint failed: mlx_configuration.variable_name[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:21 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:3009 [35;1mUNIQUE constraint failed: model_registry.environment_crn[0m
2024-02-02 22:22:28 - INFO-root::util|431:: 2024/02/02 22:21:21 [31;1m/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/db/db.go:3009 [35;1mUNIQUE constraint failed: model_registry.environment_crn[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestHasRequestData/error[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestHasRequestData/error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunBackupJob/SubmitAndRunBackupJob_should_return_error_if_drs_backup_returned_non_zero_response_code[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunBackupJob/SubmitAndRunBackupJob_should_return_error_if_drs_submitbackup_call_failed[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunBackupJob/SubmitAndRunBackupJob_should_return_error_if_drs_runbackup_call_failed[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunDeleteBackupJob/SubmitAndRunDeleteBackupJob_should_return_error_if_drs_delete_backup_returned_non_zero_response_code[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunDeleteBackupJob/SubmitAndRunDeleteBackupJob_should_return_error_if_drs_submitdeletebackup_call_failed[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunDeleteBackupJob/SubmitAndRunDeleteBackupJob_should_return_error_if_drs_rundeletebackup_call_failed[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_DescribeBackupJob/DescribeBackupJob_should_return_error_if_drs_describe_backup_job_returned_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_ListBackupJobs/ListBackupJobs_should_return_error_if_drs_listbackups_returned_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_GetJobLogs/GetJobLogs_should_return_error_if_drs_getjoblogs_returned_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunRestoreBackupJob/SubmitAndRunRestoreBackupJob_should_return_error_if_drs_submit_restore_returned_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunRestoreBackupJob/SubmitAndRunRestoreBackupJob_should_return_error_is_drs_run_restore_returned_error_[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_SubmitAndRunRestoreBackupJob/SubmitAndRunRestoreBackupJob_should_return_error_if_drs_run_restore_returned_non_zero_response_code[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_DescribeRestoreJob/DescribeRestoreJob_should_return_error_if_drs_describe_restore_job_returned_error[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestDrs/Test_PollForRestoreJobCompletion/PollForRestoreJobCompletion_should_return_error_if_drs_restore_job_failed[0m
2024-02-02 22:22:28 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"drs/drs.go:304","message":"[requestId=b4f88fe3-0a7e-4636-ba0b-a45b88bcbb6d] restore job drsRestoreBackupCrn FAILED","requestId":"b4f88fe3-0a7e-4636-ba0b-a45b88bcbb6d","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunBackupJob/SubmitAndRunBackupJob_should_return_error_if_drs_backup_returned_non_zero_response_code (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunBackupJob/SubmitAndRunBackupJob_should_return_error_if_drs_submitbackup_call_failed (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunBackupJob/SubmitAndRunBackupJob_should_return_error_if_drs_runbackup_call_failed (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunDeleteBackupJob/SubmitAndRunDeleteBackupJob_should_return_error_if_drs_delete_backup_returned_non_zero_response_code (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunDeleteBackupJob/SubmitAndRunDeleteBackupJob_should_return_error_if_drs_submitdeletebackup_call_failed (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunDeleteBackupJob/SubmitAndRunDeleteBackupJob_should_return_error_if_drs_rundeletebackup_call_failed (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_DescribeBackupJob/DescribeBackupJob_should_return_error_if_drs_describe_backup_job_returned_error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_ListBackupJobs/ListBackupJobs_should_return_error_if_drs_listbackups_returned_error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_GetJobLogs/GetJobLogs_should_return_error_if_drs_getjoblogs_returned_error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunRestoreBackupJob/SubmitAndRunRestoreBackupJob_should_return_error_if_drs_submit_restore_returned_error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunRestoreBackupJob/SubmitAndRunRestoreBackupJob_should_return_error_is_drs_run_restore_returned_error_ (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_SubmitAndRunRestoreBackupJob/SubmitAndRunRestoreBackupJob_should_return_error_if_drs_run_restore_returned_non_zero_response_code (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_DescribeRestoreJob/DescribeRestoreJob_should_return_error_if_drs_describe_restore_job_returned_error (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestDrs/Test_PollForRestoreJobCompletion/PollForRestoreJobCompletion_should_return_error_if_drs_restore_job_failed (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"error","line":"health/health.go:256","message":"[requestId=ed79db6d-ab90-4233-973e-f4440d9a5761] Visitor '&{0xc0009bbad0 0xc000566328}' encountered error 'error!'","requestId":"ed79db6d-ab90-4233-973e-f4440d9a5761","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestGetHealthInfo/KubeClient_list_nodes_failure[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestGetHealthInfo/KubeClient_list_nodes_failure/workspace_in_nonexempt_state[0m
2024-02-02 22:22:28 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"error","line":"health/health.go:201","message":"[requestId=a3527863-07ea-4407-849a-6641071fd5ca] HealthPoller: Failed to fetch nodes list from KubeAPI for workspace 'crn:cdp:ml:us-west-1:account-id:workspace:resource-name' due to error: Error listing Nodes","requestId":"a3527863-07ea-4407-849a-6641071fd5ca","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestGetHealthInfo/KubeClient_list_nodes_failure/workspace_in_exempted_state[0m
2024-02-02 22:22:28 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:altus:user:__internal__actor__","level":"error","line":"health/health.go:201","message":"[requestId=a3527863-07ea-4407-849a-6641071fd5ca] HealthPoller: Failed to fetch nodes list from KubeAPI for workspace 'crn:cdp:ml:us-west-1:account-id:workspace:resource-name' due to error: Error listing Nodes","requestId":"a3527863-07ea-4407-849a-6641071fd5ca","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestGetHealthInfo/KubeClient_list_nodes_failure (0.12s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestGetHealthInfo/KubeClient_list_nodes_failure/workspace_in_nonexempt_state (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestGetHealthInfo/KubeClient_list_nodes_failure/workspace_in_exempted_state (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestKubeImpl/CreatePersistentVolumeClaim/error_during_clienset_creation[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestKubeImpl/CreateDaemonSet/error_during_clienset_creation[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestKubeImpl/CreatePersistentVolumeClaim/error_during_clienset_creation (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::         --- PASS: TestKubeImpl/CreateDaemonSet/error_during_clienset_creation (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestPrometheusMetrics/Vault_Token_Failure_Counter[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestPrometheusMetrics/Create_Failure_Counter[0m
2024-02-02 22:22:28 - INFO-root::util|431:: === RUN   TestPrometheusMetrics/Restore_Failure_Counter[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestPrometheusMetrics/Vault_Token_Failure_Counter (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestPrometheusMetrics/Create_Failure_Counter (0.00s)[0m
2024-02-02 22:22:28 - INFO-root::util|431::     --- PASS: TestPrometheusMetrics/Restore_Failure_Counter (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestCadenceWorkflowClient_IsWorkflowClosed/Error_is_returned[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestCadenceWorkflowClient_IsWorkflowClosed/Nil_cadence_client_should_return_error[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestCadenceWorkflowClient_IsWorkflowClosed/Error_is_returned (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestCadenceWorkflowClient_IsWorkflowClosed/Nil_cadence_client_should_return_error (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestCadenceWorkflowClient_StartWorkflow/Start_a_workflow_fails_when_Cadence_returns_error[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestCadenceWorkflowClient_StartWorkflow/Start_a_workflow_fails_when_Cadence_returns_error (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestCadenceWorkflowClient_DescribeWorkflowExecution/Return_error[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestCadenceWorkflowClient_DescribeWorkflowExecution/Return_error (0.00s)[0m
2024-02-02 22:22:29 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=12f90fca-eb5b-46b5-b268-6715a1d98fd8] /api/v1/compute/createResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"FAILED","message":"internal_server_error"},"requestId":"d78adfc5-9a13-4d4a-82ad-6caca5f668e5"}","requestId":"12f90fca-eb5b-46b5-b268-6715a1d98fd8","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:338","message":"[requestId=12f90fca-eb5b-46b5-b268-6715a1d98fd8] failed to call resource pool api /api/v1/compute/createResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"FAILED","message":"internal_server_error"},"requestId":"d78adfc5-9a13-4d4a-82ad-6caca5f668e5"}","requestId":"12f90fca-eb5b-46b5-b268-6715a1d98fd8","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:318","message":"[requestId=884a6345-bbb9-400c-af31-973332a343db] failed to unmarshal POST /api/v1/compute/createResourcePool response json: cannot unmarshal string into Go value of type models.CreateResourcePoolResponse","requestId":"884a6345-bbb9-400c-af31-973332a343db","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:338","message":"[requestId=884a6345-bbb9-400c-af31-973332a343db] failed to call resource pool api failed to unmarshal POST /api/v1/compute/createResourcePool response: json: cannot unmarshal string into Go value of type models.CreateResourcePoolResponse","requestId":"884a6345-bbb9-400c-af31-973332a343db","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:99","message":"[requestId=4d7a8c0d-ac6e-41e4-bec3-6781ede647f3] failed to invoke create crp namespace request failed to invoke the crp request for dummy-namespace. error = ","requestId":"4d7a8c0d-ac6e-41e4-bec3-6781ede647f3","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=e3e9c227-8a44-42d2-b158-d9fc34825446] /api/v1/compute/listClusters crp returned status 500 with message {"clusters":null}","requestId":"e3e9c227-8a44-42d2-b158-d9fc34825446","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:194","message":"[requestId=e3e9c227-8a44-42d2-b158-d9fc34825446] failed to get cluster details from liftie:  {"envNameOrCrn":"dummyEnvName","pageSize":10} /api/v1/compute/listClusters crp returned status 500 with message {"clusters":null}","requestId":"e3e9c227-8a44-42d2-b158-d9fc34825446","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=e1305e1e-5469-47c9-bd81-ca0a8e7f570c] /api/v1/compute/updateResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"FAILED","message":"invalid resource pool"},"requestId":"a6efec83-e142-4c3b-b1fd-41b3877179df"}","requestId":"e1305e1e-5469-47c9-bd81-ca0a8e7f570c","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:164","message":"[requestId=e1305e1e-5469-47c9-bd81-ca0a8e7f570c] updateResourcePool:  /api/v1/compute/updateResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"FAILED","message":"invalid resource pool"},"requestId":"a6efec83-e142-4c3b-b1fd-41b3877179df"}","requestId":"e1305e1e-5469-47c9-bd81-ca0a8e7f570c","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=e0719a0b-4450-4ea8-b944-320ee0d78b9d] /api/v1/compute/getResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"SUCCEEDED","message":""},"resourcePool":{"path":"root.default.cml","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","policy":{"validity":"NEVER_EXPIRES","allocation":{"distribution":"INELASTIC","order":"FIFO"}},"metadata":{"tags":{"experience":"cml"}}}}","requestId":"e0719a0b-4450-4ea8-b944-320ee0d78b9d","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:118","message":"[requestId=e0719a0b-4450-4ea8-b944-320ee0d78b9d] getResourcePoolResponse:  {} /api/v1/compute/getResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"SUCCEEDED","message":""},"resourcePool":{"path":"root.default.cml","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","policy":{"validity":"NEVER_EXPIRES","allocation":{"distribution":"INELASTIC","order":"FIFO"}},"metadata":{"tags":{"experience":"cml"}}}}","requestId":"e0719a0b-4450-4ea8-b944-320ee0d78b9d","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=a86cadaa-8124-4766-a1ec-1c015438ebf8] /api/v1/compute/getResourcePoolSubtree crp returned status 500 with message {"path":"root.default.cml","status":{"state":"SUCCEEDED","message":""},"resourcePool":{"path":"root.default.cml","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","policy":{"validity":"NEVER_EXPIRES","allocation":{"distribution":"INELASTIC","order":"FIFO"}},"metadata":{"tags":{"experience":"cml"}}}}","requestId":"a86cadaa-8124-4766-a1ec-1c015438ebf8","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=9ccaebb6-5ea3-4890-a192-4d05495a6a1a] /api/v1/compute/getResourcePoolSubtree crp returned status 500 with message {"path":"root","status":{"state":"AVAILABLE","message":"dummy"},"resourcePoolTree":{"path":"root","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","childrens":[{"path":"root.default.cml","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","distribution":"INELASTIC"}],"distribution":"INELASTIC"}}","requestId":"9ccaebb6-5ea3-4890-a192-4d05495a6a1a","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:133","message":"[requestId=9ccaebb6-5ea3-4890-a192-4d05495a6a1a] getResourcePoolSubtreeResponse:  null /api/v1/compute/getResourcePoolSubtree crp returned status 500 with message {"path":"root","status":{"state":"AVAILABLE","message":"dummy"},"resourcePoolTree":{"path":"root","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","childrens":[{"path":"root.default.cml","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","distribution":"INELASTIC"}],"distribution":"INELASTIC"}}","requestId":"9ccaebb6-5ea3-4890-a192-4d05495a6a1a","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=94e86d52-cf8b-4d66-8827-53f8011e0711] /api/v1/compute/getResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"SUCCEEDED","message":""},"resourcePool":{"path":"root.default.cml","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","policy":{"validity":"NEVER_EXPIRES","allocation":{"distribution":"INELASTIC","order":"FIFO"}},"metadata":{"tags":{"experience":"cml"}}}}","requestId":"94e86d52-cf8b-4d66-8827-53f8011e0711","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:118","message":"[requestId=94e86d52-cf8b-4d66-8827-53f8011e0711] getResourcePoolResponse:  {} /api/v1/compute/getResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"SUCCEEDED","message":""},"resourcePool":{"path":"root.default.cml","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","policy":{"validity":"NEVER_EXPIRES","allocation":{"distribution":"INELASTIC","order":"FIFO"}},"metadata":{"tags":{"experience":"cml"}}}}","requestId":"94e86d52-cf8b-4d66-8827-53f8011e0711","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:142","message":"[requestId=94e86d52-cf8b-4d66-8827-53f8011e0711] getResourcePoolSubtreeResponse:  null /api/v1/compute/getResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"SUCCEEDED","message":""},"resourcePool":{"path":"root.default.cml","quota":{"cpu":"500","memory":"1000G","nvidia.com/gpu":"200"},"availableQuota":{"cpu":"480","memory":"950G","nvidia.com/gpu":"100"},"state":"AVAILABLE","policy":{"validity":"NEVER_EXPIRES","allocation":{"distribution":"INELASTIC","order":"FIFO"}},"metadata":{"tags":{"experience":"cml"}}}}","requestId":"94e86d52-cf8b-4d66-8827-53f8011e0711","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=138368aa-15e9-41e7-8b18-ae59bbd59a26] /api/v1/compute/deleteResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"FAILED","message":"invalid resource pool"},"requestId":"a6efec83-e142-4c3b-b1fd-41b3877179df"}","requestId":"138368aa-15e9-41e7-8b18-ae59bbd59a26","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:228","message":"[requestId=138368aa-15e9-41e7-8b18-ae59bbd59a26] deleteResourcePool:  /api/v1/compute/deleteResourcePool crp returned status 500 with message {"path":"root.default.cml","status":{"state":"FAILED","message":"invalid resource pool"},"requestId":"a6efec83-e142-4c3b-b1fd-41b3877179df"}","requestId":"138368aa-15e9-41e7-8b18-ae59bbd59a26","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:6ad8dfbc-9519-4de9-ab9b-e2e69f442ecf:user:ab09ab51-b5af-4b13-b73e-e8be0bde9609","level":"error","line":"resourcepool/resourcepoolmanager.go:290","message":"[requestId=8fe28cd2-962f-420f-a85b-5baa561480c8] /api/v1/compute/getRequestStatus crp returned status 500 with message {"requestId":"a6efec83-e142-4c3b-b1fd-41b3877179df","status":{"state":"FAILED","message":"Operation cannot be fulfilled on configmaps"}}","requestId":"8fe28cd2-962f-420f-a85b-5baa561480c8","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestUsageReportingActivity_TrackWorkspaceStatusChangeActivity/TrackWorkspaceStatusChangeActivity_with_no_failureErrMsg[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestUsageReportingActivity_TrackWorkspaceStatusChangeActivity/TrackWorkspaceStatusChangeActivity_with_failureErrMsg[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestUsageReportingActivity_TrackWorkspaceStatusChangeActivity/returns_error_when_TrackWorkspaceStatusChange_error[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"level":"error","ts":1706912467.436035,"caller":"usagereporting/usagereporting_activities.go:41","msg":"Failed to TrackWorkspaceStatusChangeActivity for workspaceCrn '', operation '', operationStatus '', statusChangeOptions '{    map[] {0 0         0001-01-01 00:00:00 +0000 UTC false  <nil> } {  }}': TrackWorkspaceStatusChange error","requestId":"request-id","actorCrn":"crn:partition:ml:region:tenant:actor-crn","resourceId":"resource-id","stacktrace":"github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/usagereporting.(*UsageReportingActivity).TrackWorkspaceStatusChangeActivity
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/usagereporting/usagereporting_activities.go:41
github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/usagereporting.TestUsageReportingActivity_TrackWorkspaceStatusChangeActivity.func7
	/go/src/github.infra.cloudera.com/Sense/mlx-crud-app/service/pkg/usagereporting/usagereporting_activities_test.go:106
testing.tRunner
	/usr/local/go/src/testing/testing.go:1595"}[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestUsageReportingActivity_TrackWorkspaceStatusChangeActivity/TrackWorkspaceStatusChangeActivity_with_no_failureErrMsg (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestUsageReportingActivity_TrackWorkspaceStatusChangeActivity/TrackWorkspaceStatusChangeActivity_with_failureErrMsg (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestUsageReportingActivity_TrackWorkspaceStatusChangeActivity/returns_error_when_TrackWorkspaceStatusChange_error (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetWorkspaceCreationFailureTypeFromStr[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetWorkspaceCreationFailureTypeFromStr/cluster:provision:failed[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetWorkspaceCreationFailureTypeFromStr/storage:provision:failed[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetWorkspaceCreationFailureTypeFromStr/storage:restore:failed[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetWorkspaceCreationFailureTypeFromStr/mlx_instance:preInstallValidation:failed[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetWorkspaceCreationFailureTypeFromStr/mlx_instance:installation:failed[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetWorkspaceCreationFailureTypeFromStr/unknown_failure[0m
2024-02-02 22:22:29 - INFO-root::util|431:: --- PASS: TestGetWorkspaceCreationFailureTypeFromStr (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetWorkspaceCreationFailureTypeFromStr/cluster:provision:failed (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetWorkspaceCreationFailureTypeFromStr/storage:provision:failed (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetWorkspaceCreationFailureTypeFromStr/storage:restore:failed (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetWorkspaceCreationFailureTypeFromStr/mlx_instance:preInstallValidation:failed (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetWorkspaceCreationFailureTypeFromStr/mlx_instance:installation:failed (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetWorkspaceCreationFailureTypeFromStr/unknown_failure (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=fake-request-id] {"id":"e473abfb-7150-4d2e-b7e5-4f1f76cce755","timestamp":1706912467457,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:c4d614fc-7a59-4b33-9e8a-20de062391fb","oldStatus":4,"newStatus":6,"failureReason":"your environment is bad","requestId":"fake-request-id","liftieId":"liftie-abc","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=fake-request-id] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"INSTALLATION_FAILED","OldStatus":"INSTALLATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:c4d614fc-7a59-4b33-9e8a-20de062391fb"}'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=fake-request-id] {"id":"424e61ce-7618-4974-9cc6-334681f9e04e","timestamp":1706912467458,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:c4d614fc-7a59-4b33-9e8a-20de062391fb","oldStatus":1,"newStatus":3,"failureReason":"your environment is bad","requestId":"fake-request-id"}}}","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: 2024/02/02 22:21:07 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=fake-request-id] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=fake-request-id] {"id":"4acec467-5a52-4b5a-9538-fa0c2af6dce5","timestamp":1706912467721,"Details":{"CdpCMLWorkspaceStatusChanged":{"crn":"crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:c4d614fc-7a59-4b33-9e8a-20de062391fb","oldStatus":23,"newStatus":25,"failureReason":"Failed but unit test passed!","requestId":"fake-request-id","userTags":"null","cdpCMLWorkspaceRestoreMetadata":{},"updateTags":"{\"updateType\":\"\",\"scaleRange\":\"\",\"instanceType\":\"\"}"}}}","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"info","line":"usagereporting/usagereporting.go:839","message":"[requestId=fake-request-id] Sent CdpCMLWorkspaceStatusChanged for event '{"NewStatus":"VALIDATION_FAILED","OldStatus":"VALIDATION_STARTED","WorkspaceCrn":"crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:workspace:c4d614fc-7a59-4b33-9e8a-20de062391fb"}'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: 2024/02/02 22:21:07 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=fake-request-id] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: 2024/02/02 22:21:08 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"error","line":"usagereporting/usagereporting.go:1000","message":"[requestId=fake-request-id] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: 2024/02/02 22:21:08 Failed to send provisioner status request: Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""[0m
2024-02-02 22:22:29 - INFO-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:deadbeef-1cad-45d7-b645-7ccf9edbb73d:user:13f7ce8d-2ddd-4605-afa9-b7b051a72e8b","level":"error","line":"usagereporting/usagereporting.go:865","message":"[requestId=fake-request-id] Error fetching cluster tags for usage tracking.  Liftie Id:   Continuing... Get "/liftie/api/v1/cluster/": unsupported protocol scheme ""","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:08Z"}[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_provision:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_deprovision:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_installation:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_mount:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_renew:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_suspend:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_resume:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_upgrade:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_modify:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_unmount:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_provisionRequestValidation:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_preInstallValidation:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_shutDownForBackup:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_restartAfterBackup:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_backup:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_restore:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_scaleDownForUpgrade:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_scaleUpAfterUpgrade:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_migration:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_migrationReadinessCheck:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_validateMigration:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431:: === RUN   TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_connection:failed_is_supported_or_exempted[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_provision:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_deprovision:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_installation:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_mount:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_renew:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_suspend:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_resume:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_upgrade:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_modify:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_unmount:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_provisionRequestValidation:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_preInstallValidation:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_shutDownForBackup:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_restartAfterBackup:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_backup:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_restore:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_scaleDownForUpgrade:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_scaleUpAfterUpgrade:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_migration:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_migrationReadinessCheck:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_validateMigration:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:29 - INFO-root::util|431::     --- PASS: TestGetStatusTypeFromStr_ContainsAllPossibleCombinations/validate_connection:failed_is_supported_or_exempted (0.00s)[0m
2024-02-02 22:22:30 - INFO-root::util|431:: === RUN   TestRightsChecker/SendError_when_error_is_not_GRPC_error[0m
2024-02-02 22:22:30 - INFO-root::util|431:: 2024/02/02 22:21:06 SendError:  resource not found  404[0m
2024-02-02 22:22:30 - INFO-root::util|431:: === RUN   TestRightsChecker/SendError_when_error_is_nil[0m
2024-02-02 22:22:30 - INFO-root::util|431:: 2024/02/02 22:21:06 SendError:  <nil> Permission Denied 403[0m
2024-02-02 22:22:30 - INFO-root::util|431:: === RUN   TestRightsChecker/SendError_when_error_is_a_grpc_error[0m
2024-02-02 22:22:30 - INFO-root::util|431:: 2024/02/02 22:21:06 SendError:  rpc error: code = NotFound desc = Environment was not found  500[0m
2024-02-02 22:22:30 - INFO-root::util|431::     --- PASS: TestRightsChecker/SendError_when_error_is_not_GRPC_error (0.00s)[0m
2024-02-02 22:22:30 - INFO-root::util|431::     --- PASS: TestRightsChecker/SendError_when_error_is_nil (0.00s)[0m
2024-02-02 22:22:30 - INFO-root::util|431::     --- PASS: TestRightsChecker/SendError_when_error_is_a_grpc_error (0.00s)[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:553","message":"[requestId=fake-request-id] timed out waiting to deploy pod 'pre-install-validation'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:175","message":"[requestId=fake-request-id] Dumping pod events '[{{ } {      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []} {      }  Event 1 { } 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0  0001-01-01 00:00:00 +0000 UTC nil  nil  }]'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:590","message":"[requestId=fake-request-id] timed out waiting to read validation results from pod 'pre-install-validation'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:175","message":"[requestId=fake-request-id] Dumping pod events '[{{ } {      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []} {      }  Event 1 { } 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0  0001-01-01 00:00:00 +0000 UTC nil  nil  }]'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:07Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:481","message":"[requestId=fake-request-id] Failed to delete the pre-install pod 'pre-install-validation'. Error 'failed to delete pod'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: === RUN   TestPreInstallValidationCheck/create_pod_throws_error[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:538","message":"[requestId=fake-request-id] Pre-install check pod failed to install with error 'failed to create pod'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: === RUN   TestPreInstallValidationCheck/reading_logs_from_pod_throws_error[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:576","message":"[requestId=fake-request-id] Failed to read logs from pod 'pre-install-validation'. Err 'failed to read pod log stream'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:175","message":"[requestId=fake-request-id] Dumping pod events '[{{ } {      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []} {      }  Event 1 { } 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0  0001-01-01 00:00:00 +0000 UTC nil  nil  }]'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:481","message":"[requestId=fake-request-id] Failed to delete the pre-install pod 'pre-install-validation'. Error 'failed to delete the pod'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:159","message":"[requestId=fake-request-id] failed to delete the pod","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:651","message":"[requestId=fake-request-id] Failed to unmarshall validation output to validation result. Err 'invalid character 'T' looking for beginning of value'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:461","message":"[requestId=fake-request-id] Failed to delete the pre-install persistent volume 'pre-install-check-nfs-pv'. Error 'failed to delete PV'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:461","message":"[requestId=fake-request-id] Failed to delete the pre-install persistent volume 'pre-install-check-nfs-pv'. Error 'failed to delete PV'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:159","message":"[requestId=fake-request-id] failed to delete PV","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:471","message":"[requestId=fake-request-id] Failed to delete the pre-install persistent volume claim 'pre-install-check-nfs-pv'. Error 'failed to delete PV'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:471","message":"[requestId=fake-request-id] Failed to delete the pre-install persistent volume claim 'pre-install-check-nfs-pv'. Error 'failed to delete PVC'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:159","message":"[requestId=fake-request-id] failed to delete PVC","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: === RUN   TestPreInstallValidationCheck/Read_custom_error_message_from_database_when_preinstall_pod_fails[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:553","message":"[requestId=fake-request-id] timed out waiting to deploy pod 'pre-install-validation'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:171","message":"[requestId=fake-request-id] Failed to get events for pod 'pre-install-validation' in namespace '': GetPodEvents failed","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: === RUN   TestPreInstallValidationCheck/Read_default_error_message_when_database_error_message_is_misconfigured[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:553","message":"[requestId=fake-request-id] timed out waiting to deploy pod 'pre-install-validation'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:171","message":"[requestId=fake-request-id] Failed to get events for pod 'pre-install-validation' in namespace '': GetPodEvents failed","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: === RUN   TestPreInstallValidationCheck/reading_pod_logs_causes_error_should_fail_gracefully[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:576","message":"[requestId=fake-request-id] Failed to read logs from pod 'pre-install-validation'. Err 'failed to read pod log stream'","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:193","message":"[requestId=fake-request-id] Failed to get pod 'pre-install-validation' in namespace '': GetPod failed","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431:: {"actorCrn":"fake-actor-crn","level":"error","line":"workflow/activities.go:171","message":"[requestId=fake-request-id] Failed to get events for pod 'pre-install-validation' in namespace '': GetPodEvents failed","requestId":"fake-request-id","timestamp":"2024-02-02T22:21:17Z"}[0m
2024-02-02 22:22:30 - INFO-root::util|431::     --- PASS: TestPreInstallValidationCheck/create_pod_throws_error (0.00s)[0m
2024-02-02 22:22:30 - INFO-root::util|431::     --- PASS: TestPreInstallValidationCheck/reading_logs_from_pod_throws_error (0.01s)[0m
2024-02-02 22:22:30 - INFO-root::util|431::     --- PASS: TestPreInstallValidationCheck/Read_custom_error_message_from_database_when_preinstall_pod_fails (0.02s)[0m
2024-02-02 22:22:30 - INFO-root::util|431::     --- PASS: TestPreInstallValidationCheck/Read_default_error_message_when_database_error_message_is_misconfigured (0.02s)[0m
2024-02-02 22:22:30 - INFO-root::util|431::     --- PASS: TestPreInstallValidationCheck/reading_pod_logs_causes_error_should_fail_gracefully (0.00s)[0m
2024-02-02 22:22:41 - INFO-root::util|431:: #6 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/node/manifests/sha256:cfd59e1e376ee7e806086268347ecd34e563537f73aaf891efe4a569f7546423: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit[0m
2024-02-02 22:22:41 - INFO-root::util|431:: ERROR: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/node/manifests/sha256:cfd59e1e376ee7e806086268347ecd34e563537f73aaf891efe4a569f7546423: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit[0m
2024-02-02 22:22:41 - INFO-root::util|431:: make: *** [test-web] Error 1[0m
2024-02-02 22:22:41 - [31mERROR-root::log|302:: Build command failed: error: Command "bash -c VERSION=1.43.0-b145 make test" failed with return code 2.[0m
2024-02-02 22:22:41 - [31mERROR-root::log|302:: Error computing total CPU and memory for pod docker-build-q7fzp in namespace re-jenkins: invalid literal for int() with base 10: '13561u'[0m
2024-02-02 22:22:43 - [31mERROR-root::log|302:: Error computing total CPU and memory for pod docker-build-q7fzp in namespace re-jenkins: invalid literal for int() with base 10: '13561u'[0m
2024-02-02 22:22:43 - [31mERROR-root::taskreporter_util|179:: Build failed[0m
2024-02-02 22:22:43 - [31mERROR-root::log|302:: Update build status to fail in RelEng Database[0m
2024-02-02 22:22:54 - [31mERROR-root::log|296:: BUILD SUMMARY REPORT
Overall Status: Failed
Status: Failed
If you are reading this message, it indicates that your build has failed. Here are the common errors analyzed from our script. Please investigate the command failures to find relevant error information from the LOG folder in the GBN url, there you can find info, debug and error logs
- Command "bash -c VERSION=1.43.0-b145 make test" failed with return code 2.
:: Environment Errors ::
:: Network Errors ::
- timed out
- 2024-02-02 22:22:17 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=41d9e6b7-ca56-492c-bc04-7925904f71b3] could not obtain default workload version from configuration variables in DB","requestId":"41d9e6b7-ca56-492c-bc04-7925904f71b3","timestamp":"2024-02-02T22:21:25Z"}[0m
- 2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=ddf4bb35-f4db-4dea-9e09-6446452309b5] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. provisionRequestValidation:finished status incompatible'.","requestId":"ddf4bb35-f4db-4dea-9e09-6446452309b5","timestamp":"2024-02-02T22:22:00Z"}[0m
- 2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=c7dd47c1-2992-4730-b3c6-a414cda80b11] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"c7dd47c1-2992-4730-b3c6-a414cda80b11","timestamp":"2024-02-02T22:22:09Z"}[0m
- 2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=4a532a0b-32c2-40d2-9ed2-437fbf358ae7] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"4a532a0b-32c2-40d2-9ed2-437fbf358ae7","timestamp":"2024-02-02T22:22:06Z"}[0m
- 2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a4ae9ff1-3ae1-4307-bbd9-deabf19c3929] {"id":"62d623ce-0dd3-44a0-a2d4-0bba99976891","timestamp":1706912530013,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:476e6cc3-34a0-4ead-bf50-b4ddc55f22e7","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. connection:failed status incompatible\"}],\"requestID\":\"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929\"}","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","tenantType":"PAAS"}}}","requestId":"a4ae9ff1-3ae1-4307-bbd9-deabf19c3929","timestamp":"2024-02-02T22:22:10Z"}[0m
- 2024-02-02 22:21:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"warning","line":"backup/backup_workflow.go:1591","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] error in fetching deployment cron: unexpected error","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:21:20Z"}[0m
- 2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=df6c7bba-287b-4f74-a959-9def7ebdc56f] could not obtain default workload version from configuration variables in DB","requestId":"df6c7bba-287b-4f74-a959-9def7ebdc56f","timestamp":"2024-02-02T22:20:50Z"}[0m
- 2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=58ccfa95-d9d9-45e7-a3cf-a002dadf9b30] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. provision:warning status incompatible"}],"requestID":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30"}
","requestId":"58ccfa95-d9d9-45e7-a3cf-a002dadf9b30","timestamp":"2024-02-02T22:21:53Z"}[0m
- 2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=ceb18486-5db0-4dce-b065-4aafee6e044b] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"ceb18486-5db0-4dce-b065-4aafee6e044b","timestamp":"2024-02-02T22:22:02Z"}[0m
- 2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=84fdde98-878f-462c-b0fb-aabb1816ffa2] {"id":"75df3b88-3431-45b6-a462-4eb6cf43dcde","timestamp":1706912515615,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:c108298b-211b-4907-afa1-2ea48019ecae","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. mount:started status incompatible\"}],\"requestID\":\"84fdde98-878f-462c-b0fb-aabb1816ffa2\"}","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","tenantType":"PAAS"}}}","requestId":"84fdde98-878f-462c-b0fb-aabb1816ffa2","timestamp":"2024-02-02T22:21:55Z"}[0m
- 2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:175","message":"[requestId=8190ca19-8172-4f42-8194-af600a6add6f] failed to start BackupWorkspace: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Workspace status validation","detailedMessage":"Workspace validation failed. restore:started status incompatible"}],"requestID":"8190ca19-8172-4f42-8194-af600a6add6f"}
","requestId":"8190ca19-8172-4f42-8194-af600a6add6f","timestamp":"2024-02-02T22:22:05Z"}[0m
- 2024-02-02 22:22:26 - [31mERROR-root::util|431:: 2024-02-02T22:21:07.457Z	ERROR	cluster/provision_cluster_workflow.go:147	Encountered error when checking for cluster readiness for clusterId 'liftie-1234' for resourceCrn 'resourceCrn': unexpected error	{"requestId": "request-id", "actorCrn": "actor-crn", "resourceId": "cool-modelregistry-name"}[0m
- 2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a7805472-4bef-4e3f-9cab-1c48c2ad2d77] {"id":"cd2091c7-af0a-4277-84eb-c43731c1e272","timestamp":1706912513505,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:12b81c0d-0f57-4574-9e80-5b130a828c40","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. provision:started status incompatible\"}],\"requestID\":\"a7805472-4bef-4e3f-9cab-1c48c2ad2d77\"}","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","tenantType":"PAAS"}}}","requestId":"a7805472-4bef-4e3f-9cab-1c48c2ad2d77","timestamp":"2024-02-02T22:21:53Z"}[0m
- 2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=174d5107-dc06-4d6a-b741-4a4d133966da] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"174d5107-dc06-4d6a-b741-4a4d133966da","timestamp":"2024-02-02T22:21:56Z"}[0m
- 2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=1918e957-40e0-4551-a741-959724cf60a1] {"id":"69729c4e-e62a-42a9-8a2f-9a17e8cb305e","timestamp":1706912522575,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:c914eced-88e0-4b43-8aa4-287c69bb5c6d","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. preInstallValidation:succeeded status incompatible\"}],\"requestID\":\"1918e957-40e0-4551-a741-959724cf60a1\"}","requestId":"1918e957-40e0-4551-a741-959724cf60a1","tenantType":"PAAS"}}}","requestId":"1918e957-40e0-4551-a741-959724cf60a1","timestamp":"2024-02-02T22:22:02Z"}[0m
- 2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=7442e16e-e5de-4e83-92a7-428d81366e1f] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"7442e16e-e5de-4e83-92a7-428d81366e1f","timestamp":"2024-02-02T22:21:53Z"}[0m
- 2024-02-02 22:21:22 - [33mWARNING-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:test-tenant:user:test-user-uuid","error":"record not found","level":"warning","line":"cicd/cicd.go:115","message":"[requestId=94138341-b8b0-486d-8dc8-bb3f7120da65] could not obtain default workload version from configuration variables in DB","requestId":"94138341-b8b0-486d-8dc8-bb3f7120da65","timestamp":"2024-02-02T22:20:51Z"}[0m
- 2024-02-02 22:22:20 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4631","message":"[requestId=abe713c8-508b-470f-ad00-2a681e5d8359] Pre-flight validation Category: 'FAILED', Name: 'Workspace status validation', Message: 'Workspace validation failed. installation:succeeded status incompatible'.","requestId":"abe713c8-508b-470f-ad00-2a681e5d8359","timestamp":"2024-02-02T22:21:55Z"}[0m
- 2024-02-02 22:22:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=ce36e695-0cea-4e6d-b129-8eed316e7463] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"ce36e695-0cea-4e6d-b129-8eed316e7463","timestamp":"2024-02-02T22:21:58Z"}[0m
- 2024-02-02 22:22:41 - [31mERROR-root::log|302:: Error computing total CPU and memory for pod docker-build-q7fzp in namespace re-jenkins: invalid literal for int() with base 10: '13561u'[0m
- 2024-02-02 22:22:24 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cae8d956-1513-4d18-b17f-e4dd945b80cf] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cae8d956-1513-4d18-b17f-e4dd945b80cf","timestamp":"2024-02-02T22:22:10Z"}[0m
- 2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"core/core.go:6498","message":"[requestId=571afdb8-fcc5-43f8-905a-8c3e4340793d] Workspace deprovision error handler is called","requestId":"571afdb8-fcc5-43f8-905a-8c3e4340793d","timestamp":"2024-02-02T22:21:28Z"}[0m
- 2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6] {"id":"c331f850-d8ca-4725-8559-860044cef4bb","timestamp":1706912525016,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:3b1ce31a-5c17-45c6-82dc-4499789b9a33","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. backup:succeeded status incompatible\"}],\"requestID\":\"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6\"}","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","tenantType":"PAAS"}}}","requestId":"807d6fbe-0a1d-4ecc-b3e8-3bcc045b60b6","timestamp":"2024-02-02T22:22:05Z"}[0m
- 2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2] {"id":"6b229e21-eb78-446c-b88f-ffb8a32c03ec","timestamp":1706912523936,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:11f968b3-ae43-423c-8651-ae105722d680","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. restartAfterBackup:warning status incompatible\"}],\"requestID\":\"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2\"}","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","tenantType":"PAAS"}}}","requestId":"cf49b50d-10eb-4b28-ac6a-5b1e7a8ee7d2","timestamp":"2024-02-02T22:22:03Z"}[0m
- 2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=cd256c8c-3775-45f1-b92d-d3c45fdb8da0] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"cd256c8c-3775-45f1-b92d-d3c45fdb8da0","timestamp":"2024-02-02T22:22:02Z"}[0m
- 2024-02-02 22:22:17 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=4c355724-80c8-4a69-9cc7-18971fb65d7a] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"4c355724-80c8-4a69-9cc7-18971fb65d7a","timestamp":"2024-02-02T22:21:26Z"}[0m
- 2024-02-02 22:22:18 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"error","line":"utils/utils.go:244","message":"[requestId=2512ce76-64c6-4092-b840-d4a67c159220] Failed to start ModifyClusterSecurity: Pre-flight validations failed. {"failedValidations":[{"category":"FAILED","name":"Modify cluster security request validation","detailedMessage":"'Restrict access to Kubernetes API server to authorized IP ranges' knob cannot be enabled for 'Private cluster'"}],"requestID":"2512ce76-64c6-4092-b840-d4a67c159220"}
","requestId":"2512ce76-64c6-4092-b840-d4a67c159220","timestamp":"2024-02-02T22:21:35Z"}[0m
- 2024-02-02 22:22:22 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"core/core.go:4628","message":"[requestId=c27bd676-142c-45f5-ae82-62850a6398cb] Pre-flight validations result 'FAILED'. Passed: 0, Warning: 0, Failed: 1, Skipped: 0, Total: 1","requestId":"c27bd676-142c-45f5-ae82-62850a6398cb","timestamp":"2024-02-02T22:22:03Z"}[0m
- 2024-02-02 22:22:23 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:environments:us-west-1:testAccountID:environment:testResourceID","level":"info","line":"sigmadbus/sigmadbus_mock.go:17","message":"[requestId=a16adb06-792c-4ef5-b296-b71bdc49b5fa] {"id":"3412a7c9-96a6-4a71-9e36-1aff5a28bc83","timestamp":1706912529522,"Details":{"CdpCMLWorkspaceBackupStatusChanged":{"backupCrn":"crn:cdp:ml:us-west-1:testAccountID:workspace_backup:0118144f-4c2d-4734-9edb-6977b45464af","oldStatus":7,"newStatus":9,"failureReason":"{\"failedValidations\":[{\"category\":\"FAILED\",\"name\":\"Workspace status validation\",\"detailedMessage\":\"Workspace validation failed. validateMigration:succeeded status incompatible\"}],\"requestID\":\"a16adb06-792c-4ef5-b296-b71bdc49b5fa\"}","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","tenantType":"PAAS"}}}","requestId":"a16adb06-792c-4ef5-b296-b71bdc49b5fa","timestamp":"2024-02-02T22:22:09Z"}[0m
- 2024-02-02 22:21:21 - [31mERROR-root::util|431:: {"actorCrn":"crn:altus:iam:us-west-1:000000000:user:1111111","level":"error","line":"backup/backup_workflow.go:1888","message":"[requestId=3b2caf70-61ff-4298-90ef-29aee5137bbd] error waiting for deployments to come up: unexpected error","requestId":"3b2caf70-61ff-4298-90ef-29aee5137bbd","timestamp":"2024-02-02T22:20:50Z"}[0m

 [/INST]  2024-02-02 22:22:41 - INFO-root::util|431:: #6 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/node/manifests/sha256:cfd59e1e376ee7e806086268347ecd34e563537f73aaf891efe4a569f7546423: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit[0m
2024-02-02 22:22:41 - INFO-root::util|431::  > [internal] load metadata for docker.io/library/node:12.22.6:[0m
2024-02-02 22:22:41 - INFO-root::util|431:: ERROR: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/node/manifests/sha256:cfd59e1e376ee7e806086268347ecd34e563537f73aaf891efe4a569f7546423: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit[0m
2024-02-02 22:22:41 - INFO-root::util|431:: make: *** [test-web] Error 1[0m
 </s>