What is the proper way to connect a 4" dryer exhaust to a 6" PVC wall exhaust?
Is this PVC run straight? Also, since you have posted some pictures, please post some for how the "anti bird" vent cover is mounted and secured.

Here's my thoughts:
While the previous occupant made this work, they were violating code, creating a situation that creates static electricity, and as you noted, water. You see water INSIDE the house where the vent was, but what about the wall outside where this PVC eventually meets the world? Water can be leaking in the interior walls and rotting away structure or making a nice home for carpenter ants. Not to mention the PVC (especially if there are any bends in it) is probably plugged or nearly plugged, and with an imperfect fit, harmful gasses can be present in the living space. This is not doom and gloom, but some thoughts for consideration towards "what to do".

On your question of cutting drywall; I would rather use a drywall saw and cut as needed to complete a proper run (with metal) then repair when the budget is right vs. ANY other option. Besides, you'll get a chance to see what other damage lurks in spaces unseen.

If the PVC run is straight, you can actually buy proper 4" run, and feed it through the existing PVC. While this can avoid cutting, clearly vacuum or clean out the run first, and still inspect BOTH ENDS for water damage.

If the PVC has an elbow in it, is this elbow accessible? You can always cut the PVC elbows out of the path and feed metal duct from both ends and join to an elbow when ready.

Lastly, since the run is 6" OD (outer diameter) or greater, you can probably cut sections out thoughtfully and the only immediate expense would be the ductwork *(this should be very cost effective) and perhaps a drywall saw *($7?).

Exercise some caution, secure breakers, use a studfinder/tap method, and only make the holes as large as you need. Repairing drywall afterwards can be an art but its not insurmountable and then at least you aren't wasting money on the wrong solution.