Multi User limited Time Encryption of big Datasets
I'd start by dividing the read, write, and replicate roles into 3 keys. Tahoe-LAFS does this using "semiprivate" keys, each no larger than an EC curve point. I.e. Using ed25519, these keys would all be 32-bytes. The write key implies read, and the read key implies the replicate/verification key.

Once these keys have been revealed to someone, they cannot be revoked. To implement revocation, we need an additional authority that may be revoked, such as Macaroons. Macaroons support both delegation (sharing) and provisioning (with additional restrictions). These restrictions may reference clocks, or limit the functions the user may call (I.e. to read, write, append, truncate, delete, ..), or query out to external services to either check if the credential is still valid (revocation) or to check if the message is approved by the authority (configurable filters).

You cannot force a user to delete any ciphertexts, keys/credentials, nor plaintexts. You can revoke credentials to limit the distribution of ciphertexts. You don't need to rotate the encryption key so long that your adversary cannot also obtain the ciphertext (a revoked user cannot).

If your trusted server is allowed to read the user's data, then you don't need encryption at all (aside from session transports) as you only care about the authority to access. If the users can create new documents, then you'll likely prefer to lock the server (and you, the administrator) out of the user' personal data.