German is the next-largest language represented at 9%, and is followed by a long-tail of 
languages by representation.   
In order to enable these uses, HathiTrust has invested in technical solutions to prevent 
possible misuse. To some extent, they manage this by limiting who gets access to the 
Center, and limiting access to speci ﬁc features to researchers at member institutions. 
HathiTrust has also put in place various security controls on both the physical storage of the 
digitized books and the network access to those ﬁles. The primary uses of the data through 
the Research Center includes access to an extracted features set and access to the 
complete corpus  “data capsule,” which is a virtual machine running on the Center’s servers. 
The data capsule allows users to conduct non-consumptive research with the data, but it 
limits the types of outputs allowed in order to prevent users from obtaining full content of in-
copyright works. The measures taken include physical security controls on the data centers 
housing the information, as well as restrictions via network access and encryption of backup 
tapes. In the ﬁnding that HathiTrust use was a fair use and thus rejecting a lawsuit brought 
by the Authors Guild, the Court noted the importance of these controls.  35
Today, the Center’s tools are not suitable for AI training, in that they don’t allow the speci ﬁc 
types of technical manipulation of underlying text necessary to train an AI. Nevertheless, the 
Center demonstrates that building a books data commons for computational analysis is 
possible, and in turn points to the possibility of creating such a resource for AI training.  36
Implications of Overall Approach 
By relying on existing limitations and exceptions in copyright law, the number of books one 
could include in the corpus of a books data commons is far greater and more diverse. Of 
course, a bigger dataset doesn’t necessarily mean a higher quality dataset for all uses of AI 
models; as HathiTrust shows, even a multimillion book corpus can skew in various 
directions.  Still, dataset size generally remains signi ﬁcant to an LLM’s performance – the 
more text one can train on, or rather the more tokens for training the model, the better, at 
least along a number of performance metrics.   37
While holding the potential for a broader and more diverse dataset, a key limitation in 
pursuing this approach is that it is only feasible where relevant copyright limitations and 
exceptions exist. Even then, legal uncertainty means that going down this path is likely to 
generate, at a minimum, expensive and time-consuming litigation and regulatory 
 This is explained explicitly in the appeals court’s decision: Authors Guild v. HathiTrust, 755 F.3d 87 (2d 35
Cir. 2014).
 HathiTrust has also made available some data derived from books, such as the Extracted Features 36
set: “HTRC releases research datasets to facilitate text analysis using the HathiTrust Digital Library. 
While copyright-protected texts are not available for download from HathiTrust, fruitful research can still 
be performed on the basis of non-consumptive analysis of transformative datasets, such as in HTRC's 
ﬂagship Extracted Features Dataset, which includes features extracted from full-text volumes. These 
features include volume-level metadata, page-level metadata, part-of-speech-tagged tokens, and token 
counts:” https://analytics.hathitrust.org/datasets#top.
 See Testimony of Chris Callison-Burch, July 2023, https://docs.house.gov/meetings/JU/37
JU03/20230517/115951/HHRG-118-JU03-Wstate-Callison-BurchC-20230517.pdf (“As the amount of 
training data increases, AI systems’ capabilities for language understanding and their other skills 
improve.”); Brown, Tom, et al. Language Models Are Few-Shot Learners. 22 July 2020, at https://arxiv.org/
pdf/2005.14165.pdf (“we ﬁnd that performance scales very smoothly with model size”).
Towards a Books Data Commons for AI Training 15