Un-optimize a MySQL database?
No.  Do not expect much, if any, performance boost with OPTIMIZE TABLE.

ANALYZE TABLE is slow on MyISAM, the engine you should be moving away from.  It is fast in InnoDB.  It is almost never needed in either engine.  Only if you have a particular query that defies rational analysis by a human, should you consider ANALYZE.

OPTIMIZE TABLE will rebuild the table.  That requires blocking all writes to it for the duration.

Back to your final question...  Dumping the data (not the files) will provide you with something that can be loaded into another server for the testing you suggest.  However, the dump, itself, involves some degree of locking.  This depends on which engine you are using and whether you need a "consistent" dump.

SHOW TABLE STATUS will give you a clue of Data_free.  For MyISAM, that is how much space can be recouped with OPTIMIZE.  But even a large number there does not necessarily mean much performance boost.  For InnoDB, Data_free has less meaning and its meaning depends on the value of innodb_file_per_table when the table was created.