Passing around context.Context in Golang
You've actually improved the design, because now unit tests can inject their own context and thus the shutdown() becomes testable. Pure gold.

But your stated reasons make me wary. I'd pass a parameter (e.g. struct) to Start() or NewServer()  There aren't many good uses of WithValue that make your code more readable and neater.

Also docs for WithValue warn that "The provided key must be comparable and should not be of type string or any other built-in type to avoid collisions between packages using context." So that's a big problem with your example.