Home video, Canon 60D, 1080p, 23 fps, H.264 lag during playback
This is almost certainly a data rate problem.  The quality level of H.264 from a camera is much higher and much less compressed than a typical media file.  

H.264 is an elaborate compression which can achieve substantial compression ratios while maintaining high quality, however it is also very complex and difficult to encode with the best compression settings available.  To save time and battery, a camera such as the 60D uses a hardware encoder and applies only simple compression.  This means the file size is much larger (though the quality is also higher).

Before final consumption, the video file should be re-compressed to a more compressed file for final viewing.  You can use any of a number of compressors (either one built in to a video editing package of your choice or you can use the free FFMpeg).  If you use a two pass VBR preset for something like YouTube it should encode to a much more highly compressed and network/low power device playback friendly compression.  If you have an option for profile, I'd try the low profile as it is supported by more low power devices than the high or medium profiles.