Virtual ICCM I

Understanding the fundamental cognitive process of decision-making is crucial for developing appropriate cognitive models. Two main planning-based approaches have been used to investigate learning in complex decision-making tasks: one using model-based reinforcement learning, an extension of reinforcement learning that includes high-level planning, and the other using instance-based learning (IBL), based on episodic memories of previous interactions. In this paper, we attempt to reconcile the two approaches by using ACT-R to implement a cognitively plausible substrate for the planning component of MB-RL. We review the model-based (MB) and model-free (MF) learning approaches in reinforcement learning and discuss their roles in decision-making strategy. Within the ACT-R framework, we propose a promising model that incorporates memory retrieval in MB planning, offering a cognitively plausible approach to the planning component of MB-RL. Our combined model successfully replicates well-known findings in the literature, including developmental reliance on memory and response time variations between common and rare options. Finally, our model naturally accounts for the balance of memory and RL depending on the relative cost of each. We argue for the superiority of our cognitive model and address the significance of this study for understanding the brain and computational processes underpinning decision-making strategies, as well as for applications in artificial intelligence and decision-making modeling.

Learning from experience, often formalized as Reinforcement Learning (RL), is a vital means for agents to develop successful behaviours in natural environments. However, while biological organisms are embedded in continuous spaces and continuous time, many artificial agents use RL algorithms that implicitly assume some form of discretization of the state space, which can lead to inefficient resource use and improper learning. In this paper we show that biologically motivated representations of continuous spaces form a valuable state representation for RL. We use models of grid and place cells in the Medial Entorhinal Cortex (MEC) and hippocampus, respectively, to represent continuous states in a navigation task and in the CartPole control task. Specifically, we model the hexagonal grid structures found in the brain using Hexagonal Spatial Semantic Pointers (HexSSPs), and combine this state representation with single-hidden-layer neural networks to learn action policies in an Actor-Critic (AC) framework. We demonstrate our approach provides significantly increased robustness to changes in environment parameters (travel velocity), and learns to stabilize the dynamics of the CartPole system with comparable mean performance to a deep neural network, while decreasing the terminal reward variance by more than~150x across trials.These findings at once point to the utility of leveraging biologically motivated representations for RL problems, and suggest a more general role for hexagonally-structured representations in cognition.

Individual learners rely on different strategies - combinations of declarative and reinforcement learning - to acquire new skills. But little is known about how these strategies change throughout the duration of learning. In this study, we use four idiographic ACT-R models to fit and identify learning strategies during a stimulus-response learning task (Collins, 2018). We split the long learning task into two halves and fit independently to address this. We found that a majority of learners relied on declarative memory (LTM) throughout learning. Of the minority of learners who were identified by a reinforcement learning strategy (RL) or combined RL-LTM strategy were more successful in the second half, if they fit an LTM only strategy.

Recent studies suggest that errors facilitate learning in certain conditions. Despite this, reinforcement paradigms dominate learning methods, subscribing to the narrative that errorless learning is the foundation of an ideal learning environment. If we continue to view learning from this restrictive perspective, we may fail to capture and apply the benefits of errors. In this paper, we investigate two potential mechanisms of learning from errors. Participants (N = 61) learned word pairs in either a study or error trial before taking a final test. Supporting past error learning literature, errors before a study opportunity led to better performance on a final test. Differences in reaction times between conditions support the theory that errors increase learning by acting as a mediator, or secondary cue, to the correct answer on subsequent tests. Individual differences in model fit using log-likelihood trial-by-trial calculations solidified support for the mediator method.

Memory errors caused by factors such as personal attributes and emotional states can be classified into two main types: errors of commission (wrong item recollection) and errors of omission (failures of recollection). In this study, these errors were obtained through an online experiment using a crowdsourcing site, and their consistency with the ACT-R model was examined. The results showed that the estimated model parameters for individual participants correlated with several reported emotional states and that it is possible to estimate the human internal state from the expressed memory errors.