Can I carve a portion out of the side of a floor joist to accommodate a Fernco Clamp
Usually we find plumbers notching the bottom of joists, not “trimming” the sides of joists. 

Structurally we check for three things: 1) bending, 2) shear, and 3) deflection. 

1) Bending is critical in the middle third of the joist. You indicated that the side trimming will occur close to the end. So, bending will not be an issue. 

2) Horizontal Shear generally occurs with large concentrated loads on SHORT spans. That is to say, normal joists spaced evenly across a floor, usually fail because of bending or deflection not shear, (unless you have a refrigerator or water bed located directly above this particular joist, I doubt if horizontal shear is an issue.)

Vertical shear occurs when large loads are imposed on joists at the bearing point. Usually this becomes evident when the the joists start crushing. For this to happen, you’d have to have a very large load (like a post carrying a large roof load) directly on the joist. 

3) Deflection is a problem with longer spans (in your case I’d guess 20’ to 22’ span or longer.) When joists deflect too much they will first crack the gypsum board on the bottom of the joists and will first become evident in the middle third of the span. You’ll also “feel” the floor bouncy or spongy when you walk across it. In your case, I doubt if you’ll notice a difference. 

The Code does not have a “rule of thumb” or limit to the amount of side trimming on any one joist...it just requires you to calculate the strength. Fortunately, the code allows an increase in stress values when joists are used repetitively. So, I’m sure you’ll be fine if you trim no more than 1/2” off the side of a 2x12 near the bearing end.