Should Amazon RDS SQL Server 2017 Web Edition Max Memory settings be different from that of a locally installed instance? We have an Amazon RDS SQL Server Web Edition on Windows Server 2012 with 2 processors and 8GB RAM.
Should I configure the Max Memory similar to that of an instance on a local server i.e keeping back 3-4GB for OS, memory pool etc. or does RDS require a different calculation?
No, I wouldn't treat this any differently than you would an on-prem instance with regards to max server memory.  The Windows Server 2012 box is almost certainly a VM, but hopefully Amazon is a managing that well to avoid VM problems.

I would definitely reserve 3 GB at minimum for the OS and other overhead.