To guide materials design and optimization for targeted vacancy defect diffusion properties, first-principles calculations of defect migration activation energies based on nudged elastic band (NEB) calculations are often utilized to provide critical atomistic insights. However, they become extremely expensive due to their reliance on density functional theory (DFT), particularly due to the need for large host supercell structural representations to avoid spurious interactions between the periodic defect images. Even more difficult is the prediction of, for example, temperature-dependent vacancy diffusivity tensors, because one must compute the vacancy migration activation energy of all plausible symmetrically inequivalent paths, of which there can be many for compositionally or structurally complex materials. To address the large computational cost of these calculations, surrogate modeling approaches like cluster expansion (CE) have been employed to predict defect migration activation energies in a variety of contexts for specific types of structures and chemistries. Meanwhile, other approaches to mitigate this difficulty have focused on a priori estimation of low energy paths to lower the number of first-principles calculations needed subsequently. Substantial work has recently been devoted to the use of various machine learning surrogate models for vacancy or interstitial defect thermodynamic properties, and defect migration activation energies have been well correlated with physical descriptors. Here, we extend these efforts and derive generalized structure-property surrogate models of NEB-calculated defect migration (i.e., yielding both formation energies and activation energies) that can be applied across any structure type or chemistry space with an accuracy that is principally limited by quantity and diversity of NEB training data. Critically, the only input to the model is the relaxed host structure and a tuple of indices that represent the start and end atomic sites for the vacancy migration event; i.e., relaxed structures for each NEB image are not used. The basis for this model is a graph neural network architecture, whose convolution functional form could in principle be taken from one of the many variants recently proposed, which is then hybridized with a transformer encoder to enforce various physical, NEB-required symmetry constraints in the model's output.