Could a large group of miners theoretically lower the network hashrate to manipulate difficulty? I am relatively new to understanding the fundamentals of the Bitcoin network, so I am still trying to learn how the actions of miners can influence the direction of the blockchain.

Back to my main question, since the difficulty adjusts at a fixed rate of every 2016 blocks (~2 weeks), what would prevent a large hashrate miner or pool from entirely shutting down their mining power minutes before the 2016th block was found? The incentive here would be that the next difficulty would become significantly lower until it is readjusted in 2016 more blocks. This process of course, could be repeated.

Is there some sort of method in place which averages transaction times preventing this from happening? If so, would such an action still have any affect?

Thanks :)
The difficulty adjusts every 2016 blocks based on the average block time for all of those 2016 blocks. So just making the 2016th block take longer to mine has little effect on adjustment. A miner would have to manipulate block times for the entire 2016 block period to get any significant difficulty change.