Is it safe for me to switch from an 8GB server to a 4GB server? (Stats inside)
It's possible. You will potentially lose capacity to serve all data requests from in memory buffers but that depends on how large your data is. 

select 
ROUND(SUM(data_length+index_length/pow(1024,2))) as TotalSizeMB, 
SUM(table_rows),
engine 
from information_schema.tables
where table_schema not in ('mysql','information_schema')
group by engine

to see your distribution per engine. You'll be able to judge whether all data and indexes fit inside the innodb bufferpool or indexes within the myisam key buffer. Assuming you set these buffers up yourself in the my.cnf config file and you're not running with the defaults.