The delay is specified from the start of the packet being transmitted at the source to the start of the packet being received at the destination.
Instantaneous packet delay variation is the difference between successive packets-here RFC 3393 does specify the selection criteria-and this is usually what is loosely termed "jitter", although jitter is also sometimes the term used for the variance of the packet delay.