Worth the Weight: Integration of Verbal and Numeric Information in Graduate Admissions

In graduate admissions, as in many merit-based decisions, evaluators must judge candidates from a flood of information, including recommendation letters, personal statements, grades, and standardized test scores. Some of this information is conveyed numerically, while some is conveyed verbally. This creates a challenge for studying these decisions, as most theories of behavioral economics specifically focus on evaluating decisions using only verbal or numeric information – not both. The goal of this study is to evaluate how verbal and numeric information are used within graduate admissions decisions. We examine a uniquely comprehensive dataset of 2,231 graduate applications to the University of Kansas, containing full application packages, demographics, and final admissions decisions for each applicant. To make sense of our documents, we apply structural topic modeling, an extension of correlated topic modeling which allows topic content and prevalence to covary based on other metadata (i.e. department of study). This allows us to examine not only what information letters and statements contain, but also the effects of gender, race, and department on how that information is conveyed. We find that admissions decision committees behaved as if they prioritized numeric metrics, using verbal information to check for disqualifications if at all. Furthermore, we find that applicant race and gender influence the prevalence of topics in their letters and statements.