Woolley was married and sealed to his first wife, Julia E. Sirls, in March 1851 and was endowed at the same time.
Woolley was uncle to LDS Church President Spencer W. Kimball, and apostles J. Reuben Clark and John W. Taylor.