Views event listing by date with up/down infinite scroll
You can achieve your requirement by following below steps.


Install Views Field View module.
Create first view of Event content type with 2 events which are previous to the current date
Create second view of Event content type with 2 events which are greater than current date.
Now create on third view with current date and into its header part add first view and into footer part add second view using Views Field View. Follow screenshot attached below 


 

Regarding scroll down functionality install Views Load More module. 

I don't know how to achieve scroll up functionality. It something like you have to write custom thing.