It wasn't idleness (a vice) but leisure (Greek skole, from which, the Latin scola and English "school) that the Ancient Greeks and the Christian monastic tradition regulated by St Benedict opposed to what we today call work. 
The "ora" in his motto is uttered within a  contemplative tradition that draws on Aristotle's conception of leisure as a state of receptiveness and interior freedom essential essential for its practice. The "labora" to which it is joined is service and participation in God's work of creation, founded on contemplation and inextricably joined to it. 
The issue isn't so much balance but integration. St Ignatius of Loyola would later offer another synthesis of ora et labora, articulating an ideal for a more active engagement of the secular order by "contemplatives in action" (whose "action" is again different from  our "work.")  
None of them would have understood our antinomy between work and leisure, nor what those words now mean. They had a conception of human nature and of the meaning of human existence that is illegible within the modern meaning of those terms.  
See, of course, Joseph Pieper's Leisure, The Basis of Culture (1952)