Views - offset when displaying all results?
I'm pretty sure that's the way it's designed.  If you set the number of things to be displayed to display all results or 0 (which is the same thing), it doesn't make sense to have an offset because you already ordered it to show every single result.

If you want to use an offset, set a specified number of results.  If you don't know what number to set, just pick an arbitrarily high number that you are unlikely to exceed.

EDIT: If you just want to hide the first item, is there some way to configure one of the filters so that the item is removed?  That is another possible approach.