How are insulation R-values calculated for wall cavities of inconsistent thickness?
The Building Code (Energy Code) stipulates the R - values allowed for various building components. (See ICC Chapter 11)
Likewise the Energy Code stipulates the maximum amount of windows, doors, skylights, etc. allowed. (See Table N1104.1(1) and Table N1104.1(2))
All calculations are based on the sum of the R - values in any particular component, (i.e.: walls, roof, etc.) To answer your question: when the component varies in thickness, then the AVERAGE of the R - value for any given AREA (given in square footage) is used.
The Energy Code gives allowable “Prescriptive Paths” that meet the Energy Code criteria for walls, slabs, skylights, HVAC, windows, roofs, etc. If for any reason your construction does NOT meet the “Prescriptive Path” for any component, then you can substitute a higher rating for another component...provided it meets or exceeds the rating and area. (It gets a little tricky substituting a higher rated HVAC unit for using more windows than what’s allowed by the prescriptive path, but there is a formula based on your heating / cooling degree days.)
There are energy calculators you can download, which are suitable for your area (zone).  You can stipulate if the studs are 16” oc or 24” oc,I don’t think you can automatically do “no studs”, but you could add the R - values up longhand. I’d call your local Building Official to get the name of such a software program. (Even I can use the program and I have trouble adding 2 + 2)