Determining if a Protein Model Contains a Backbone Clash
The way to check for steric clashing between any two atoms, backbone or otherwise, is to compute their Euclidean distance. If a and b represent two atoms (with a_x being the X coordinate of atom a and so forth), you can calculate their Euclidean distance as follows.


  d(a, b) = sqrt( (a_x - b_x)^2 + (a_y - b_y)^2 + (a_z - b_z)^2) )


So essentially the idea would be to calculate the pairwise distance between each of the backbone atoms. For any pair of atoms, there is steric clashing if the distance between them falls below a certain threshold. If I remember correctly, this threshold is the sum of the van der Waals radii of the two atoms.