What field is Views using to determine the pager 'offset'?
It offsets by row. 

If offset = 5, it will skip the first 5 rows of your view result.

By default, the View SORT is using Post Date. 

Remove Post Date and add the desired field to the SORT area. (when adding, look for your custom date field that you created.)