Which are the downsides of not deleting the CDC records in change tables?
What does 'grow unmanageable' means for me?


This all depends on your environment. If you had a DB that was subject to a lot of transactions (one with a lot of CRUD operations), then this table would grow much faster and larger than one with minimal changes making it harder to manage.  


  Will it have very negative impact on the performance or storage in the
  DB? Maybe an impact on Replication performance?


The larger your change tables grow, the longer your ETL / change data query functions that are used to store the data on another instance would take to run (they would be slower). You haven't stated of you are storing these records elsewhere, or just using the change tables as your end point for auditing, but it seems like the latter. 


  Am I just fine doing CDC with no cleanup job at all?


If your change table isn't expected to grow > 20K rows ever you probably wouldn't see a performance burden.


  Is it a better alternative to copy the data to a reporting DB using a
  SSIS package or a job running a stored procedure? If so, is there a
  simple way to handle the creation of new columns in the destination
  table based on new columns added in the change tables?


Based on this question, and the fact that you state you already have a script that is copying data to another server, it seems like you are trying to capture changes multiple ways. One way you may want to explore is System-Versioned Temporal Tables. This may be the easiest route for you and could eliminate CDC and your SSIS / custom script especially since your DML events seem to be low. These tables will also change when you conduct DDL changes to your source table. 


  Does the answer change if instead of 20K rows in one change table
  there are 1 million?


Since that would be 50X the number of rows, sure, you can expect different results regarding performance of any queries against those change tables.