Why don't most military transports use modern commercial engines?
Some do; the C-17 you mention uses the Pratt &amp; Whitney PW2000 which is also used on the Boeing 757. The USAF and USN also fly the C-40 which is the military variant of the 737 and carries the same engines. 

Generally the answer is that military aircraft are built on spec for the military which is a bit different than the way the civilian market works. Civilian builders shop a design around to airlines to get some deposits then build the airframes (the notable exception being the 747 which was largely the idea of Pan-Am CEO Juan Trippe). This often causes there to be new aircraft every few years on the civilian side as Boeing needs to stay in business... On the contrary the military will put out a request for designs to meet their needs, evaluate the submissions and chose what is to be built. They can do this as often or as little as they so please (or as budget allows). If there is no active conflict or need for new tech then the military will keep flying what they have. Many military aircraft have cutting edge tech from the era they were designed in, they may however be dated by todays standards.

Much like the FAA here in the US the various military branches have their own certification programs and regulations thus upgrading and bringing in new gear may require something to be approved this can unfortunately stifle new technology being included due to high time and cost of approval. Similarly military requirements may be different, while a civilian airframe may suffice they may have takeoff length requirements or something similar that necessitate more thrust than their civilian counterpart thus variances in outfitting may occur.