Why does the TLS spec for TLS-PSK say not to derive the shared secret from a password? According to a note at the end of section 2.2 of the TLS 1.3 spec (RFC 8446),

When using an out-of-band provisioned pre-shared secret, a
critical consideration is using sufficient entropy during the key
generation, as discussed in [RFC4086].  Deriving a shared secret
from a password or other low-entropy sources is not secure.  A
low-entropy secret, or password, is subject to dictionary attacks
based on the PSK binder.  The specified PSK authentication is not
a strong password-based authenticated key exchange even when used
with Diffie-Hellman key establishment.  Specifically, it does not
prevent an attacker that can observe the handshake from performing
a brute-force attack on the password/pre-shared key.

Is the vulnerability of concern here that observing a handshake would allow an attacker to brute force the shared secret offline based on the PSK binder?
And, if an attacker could recover the shared secret, is confidentiality of the previously-observed TLS-PSK stream compromised? Or would this only allow the attacker to compromise future streams (e.g. by spoofing the server and/or client as a man-in-the-middle?)
Is the vulnerability of concern here that observing a handshake would allow an attacker to brute force the shared secret offline based on the PSK binder?

Sometimes; there are two ways for TLS 1.3 to negotiate with a PSK; with the PSK only, or with a PSK and an (EC)DH.  If an (EC)DH is used, the vulnerability of concern here is that a client (who does not initially know the PSK) would be able to attempt a negotiation with a server, and as a result of that exchange, run an offline test on a large number of possible PSKs.  On the other hand, if only a PSK is used, then yes, a passive attacker could go through and test the various PSKs (and then decrypt the encrypted data).
When (EC)DH is used, then what an attacking client would do is send the client hello (with the appropriate pre_shared_key identifier, and an honest DH key share, that is, one for which he knows the private value); the server would respond with a server hello, followed by &quot;encrypted extensions&quot;.
This encrypted extension is encrypted with a key which is based on stuff the client knows (nonces, the DH shared secret), and the PSK.  What the client can do, for each guess of teh PSK, plug that into (with the inputs he already knows) into the TLS 1.3, and derive the corresponding session key, and then use that key to verify the authentication tag in the encrypted extension.
Once he finds a PSK that makes the authentication tag validate, he can then use that PSK to masquerade as either the client or the server in future exchanges (or do an active MITM).

Or would this only allow the attacker to compromise future streams (e.g. by spoofing the server and/or client as a man-in-the-middle?)

TLS 1.3 allows either PSK-only mode, or a PSK along with an (EC)DH exchange.  If a PSK-only mode is chosen, then the attacker can passively listen in.  If the client decides to do PSK plus (EC)DH, then the attacker cannot passively listen into future (or past) streams; this allows only active attacks.