Research about geometry negotiation in widget layout
I would suggest looking at these survey papers and tracking down the citations in the bibliography, and digging up subsequent papers which cited these. 


Walter Hower, and  Winfried Graf. Research in constraint-based layout, visualization, CAD, and related topics : a bibliographical survey, 1995. 
Simon Lok and Steven Feiner, A survey of automated layout techniques for information presentations, 2001. 


If you're more interested in the engineering aspects, you should look at Brad Myers' work (from the late 80s/early 90s) on the Garnet and Amulet toolkits. They have a TOPLAS paper, Lessons learned about one-way, dataflow constraints in the Garnet and Amulet graphical toolkits, which summarizes what they learned.