Assessment

In graduate admissions, as in many merit-based decisions, evaluators must judge candidates from a flood of information, including recommendation letters, personal statements, grades, and standardized test scores. Some of this information is conveyed numerically, while some is conveyed verbally. This creates a challenge for studying these decisions, as most theories of behavioral economics specifically focus on evaluating decisions using only verbal or numeric information – not both. The goal of this study is to evaluate how verbal and numeric information are used within graduate admissions decisions. We examine a uniquely comprehensive dataset of 2,231 graduate applications to the University of Kansas, containing full application packages, demographics, and final admissions decisions for each applicant. To make sense of our documents, we apply structural topic modeling, an extension of correlated topic modeling which allows topic content and prevalence to covary based on other metadata (i.e. department of study). This allows us to examine not only what information letters and statements contain, but also the effects of gender, race, and department on how that information is conveyed. We find that admissions decision committees behaved as if they prioritized numeric metrics, using verbal information to check for disqualifications if at all. Furthermore, we find that applicant race and gender influence the prevalence of topics in their letters and statements.

This is an in-person presentation on July 20, 2023 (09:00 ~ 09:20 UTC).

Raven-like matrices are widely used to evaluate human intelligence and abstract reasoning. However, few resources are available for automatically generating them. Some of these resources (e.g., Corvus) are hardly customizable unless one has medium-high expertise in JavaScript, while others (e.g., the IMak package in R) are mostly focused on figure analogies based on the rotation of different objects. The ideal solution would be an open-source and easy-to-use software that implements different sets of rules for the automatic generation of Raven-like matrices. This talk presents “matriKS”, an R package for the automatic generation of Raven-like matrices, available on GitHub at https://github.com/OttaviaE/MatriKS. The package implements different sets of rules, from the most basic ones (i.e., visuo-spatial rules like changes in size and/or orientation) to the most complex ones (i.e., logic rules based on inferential and inductive reasoning), and allows the users to concatenate them with different directional logics (i.e., horizontal, vertical, diagonal logics). Different matrices have been generated with the matriKS package and they have been administered to a sample of Italian children (age 4-11). Validation of the matrices has been conducted via Rasch model analyses and it also considered the rules used for generating them and the different schooling levels.

This is an in-person presentation on July 20, 2023 (09:40 ~ 10:00 UTC).

Cognitive tutors typically use a student model to track progress of the learner. This model can be used to give feedback to teachers and students, and to select new material and assignments. Student models are typically constructed by modelers and/or education specialists. However, it is hard to assess whether the constructed student model aligns with knowledge and skills students actually need to master the material.
Instead, we propose a hybrid approach, in which we use bottom-up machine learning methods to use individual differences in student performance to construct a knowledge graph, in which each node represents a possible knowledge state of the student.
As a pilot, we constructed a knowledge graph for an arithmetic course in the mid-level vocational education (MBO) in the Netherlands. The basis for this graph was an math entry test, which, according to the publisher, addressed several specific topics, such as length measurements, weight, clock time, etc. However, when we constructed a knowledge graph from data from 413 students, we found that students do not differ on mastery of those topics, but rather on more general underlying skills, such as general arithmetic skills, reading skills and multi-step reasoning. A pilot conducted in two schools using a dashboard representing the knowledge graph was judged to be insightful and helpful by both teachers and students, and can serve as a basis for the construction of a cognitive tutor.

This is an in-person presentation on July 20, 2023 (10:00 ~ 10:20 UTC).

Unfolding models are relevant in all cases when respondents set their agreement levels by searching some optimal level of agreement with an item: They agree to some extent, but not too much. For instance, the more we see the negative consequences of having a baby for a single mother, the more we are likely to find pros about abortion. But the more we would raise our level of agreement in favor of abortion, the more we would be concerned that lives are being stopped, and this would act as a moderator of the first concern. In this situation, responses are shaped by the particular equilibrium each respondent finds between a social concern and a natural concern for life respect. In this talk, we are interested in the general class of situations where an increase in some attitude or behavior A triggers an increase in another attitude or behavior B, that at some point, eventually becomes an inhibitor of the very process that first gave it birth. This mechanism is expressed as an explicit set of differential equations, which, upon integration, leads to a new class of potentially asymmetric unimodal response functions. The obtained solution function is integrated within a Beta Response Model (Noel & Dauvier, 2007; Noel, 2014), which properties are studied, in particular by comparison of previous proposals, and an application on a real dataset is presented and discussed.

This is an in-person presentation on July 20, 2023 (10:20 ~ 10:40 UTC).