Are LLMs the end of online studies?

Due to the recent advances in artificial intelligence and large language models (LLMs), applications that involve (semi-) autonomous agents and reasoning capabilities became not only possible, but also increasingly accessible for a broader user-base. While the reasoning capabilities of LLMs are a thriving field of research, a problem for human cognitive science research arises: Studies performed online can now be solved by AI agents more easily and to a much greater extent than ever, with attention checks and captchas being less of a hurdle to bots. We investigate the issue based on a variety of reasoning tasks that have previously been used in online experiments. Using the modern agent capabilities of LLMs to interact with web browsers, we tested their ability to participate autonomously in studies. We instructed them to behave human-like, simulating how actual users might use them to participate on their behalf. Our work focuses on three key questions, namely (I) how well the agents can understand and navigate through the studies from a technical standpoint, (II) if the resulting study data is easily recognizable and distinguishable from human behavior, and (III) how accessible the respective agents are in terms of cost and effort to set them up. We test ChatGPT and Claude as two of the best LLMs for agent usage but also include models running locally on PC hardware. The results are critically discussed focusing on the current implications for online studies, but also an outlook on how to handle this issue in the future.