Literature suggests that 311 data alone may not be sufficient to measure collective efficacy and similar social constructs, rather big data combined with individual surveys and qualitative data are a more effective measure of behavioral constructs.
Daniel O'Brien (2015) examines the use of 311 data as a measure of custodianship in Boston. Custodianship is a concept similar to collective efficacy that refers to the practice of an individual taking ownership of repairing physical disorder in public space. O'Brien pulled data from the Boston system regarding public maintenance such as graffiti and potholes. Findings were that 311 databases and similar government service hotlines are useful resources for providing insight on custodianship as a community engagement process.
The author notes consideration should be taken when assessing motivation for reporting disregard in public space. Incivility reports and property neglect were overwhelmingly concentrated within narrow geographic range of callers’ property. Analysis found homeowners three times more likely to report issues compared to renters and eighty percent of reports were within two blocks of the caller’s residence. Such patterns are more indicative of territorialism and not necessarily custodianship. Constructs like custodianship generally requires evidence of larger-scale, collective community investment (O’Brien 2015).
O’Brien examines the discriminant validity of using 311 data as collective efficacy measure. Data analysis examines similarities in call proximity and report type to gauge the convergence necessary to suggest collective efficacy versus territoritorility. Data collected from 311 and similar systems are data regarding individual behavior. Authors maintain that 311 data only cannot inform all motivation.
In other research, O’Brien and colleagues (2015b) assess data from 311 hotlines as a method of gauging the plausibility of the broken windows theory. The article “Ecometrics in the Age of Big Data: Measuring and Assessing ‘Broken Windows’ Using Large-scale Administrative Records” analyzes 300,000 calls to Boston’s constituent relationship management (CRM) system over a 16-month period. The constituent relations management system (CRM) is Boston’s equivalent of a 311 hotline. System data were found to be an adequate indicator of civic response to issues such as private neglect as well as public denigration. Researchers found use of the measure detailed and cost effective.
In the article “The Effect of 311 Calls for Service on Crime in D.C. at Microplaces” Andrew Wheeler also studies the plausibility of the broken windows theory of crime by examining the relationship between crime levels and 311 calls in Washington, D.C. The broken windows theory argues that attending to the upkeep and appearance of communities communicates lower crime acceptability in a neighborhood, thus reducing opportunities for crime. Wheeler hypothesizes that communities with higher levels of 311 calls will also have lower crime rates.
Data collected on 311 calls suggest that calls for service are a valid indicator of physical disorder in a community. Wheeler concludes that 311 calls may signify community investment and high levels of collective efficacy in a neighborhood as individuals invested their community are more likely to report signs of disorder. However, the author also notes that higher crime rates may precede increased 311 call rates, and higher crime rates may indicate lower collective efficacy.
In “The Promises and Pitfalls of 311 Data” Ariel White and Kris-Stella Trump look at 311 service request lines as a viable and non-self-reported measure of citizens’ interactions with government. Their research compares 311 data to various forms of civic participation such as voter turnout, political donations, and census return rates. Their findings were that 311 calls are negatively related to lower-cost markers of civic engagement such as voter turnout and census return rates but positively related to the high-cost markers of civic engagement such as donations to political campaigns. Authors are ambivalent about using 311 data to measure community engagement, but find data potentially useful for developing an understanding of neighborhood conditions and service needs.
The authors also cite previous research that identifies census mail returns as a strong measure of community cohesiveness and other potential markers of collective efficacy. Researchers compare those data from census mail participation to 311 calls. They found that when compared to 311 calls, census return rates “better conceptualized non-political but civic-minded and public-good oriented behavior.”
Census data is another commonly used method to measure constructs such as collective efficacy and community engagement. D. Martin and colleagues in their article “Measuring Aggregate Social Capital Using Census Response Rates” examine the effectiveness of using census participation as a social capital measurement. Social capital is a concept similar to collective efficacy in that it examines the impact and importance of civic and social engagement within a community. Researchers found Census data appealing due to its cost-effectiveness and improved reliability when compared to self-reported surveys on social capital. Surveys are often vulnerable to higher levels of nonresponse bias and social desirability bias. Researchers also note that community response rate data have no risk of sampling error.
On its own, 311 data is generally insufficient to measure collective efficacy and similar constructs such as social control and territorialism. Large systems data is generally more informative when combined with survey data in order to assess the motivations for the reports. Hotlines may not be adequate measures of complex constructs as they were not created to measure collective efficacy or similar constructs such as social control. (O’Brien 2016)
There are several factors besides collective efficacy that influence the likelihood of reporting to systems such as 311. Desire to develop and uphold social norms that encourage community collaboration is a possible motivation for using CRM systems to address incivilities, however protecting and improving property values, without regard to community building is also a recognized motivation for making reports (O’ Brien 2014, W. Seo and von Rabenau 2011).