Theory Is All You Need: AI, Human Cognition, and Causal Reasoning Teppo Felin, a,b, * Matthias Holweg b a Huntsman School of Business, Utah State University, Logan, Utah 84322; b Saïd Business School, University of Oxford, Oxford OX1 1HP, United Kingdom *Corresponding author Contact: teppo.felin@usu.edu, https:/ /orcid.org/0000-0003-2044-0145 (TF); matthias.holweg@sbs.ox.ac.uk, https:/ /orcid.org/0000-0001-9403-1681 (MH) Received: February 23, 2024 Revised: April 20, 2024; July 3, 2024 Accepted: September 4, 2024 Published Online in Articles in Advance: December 3, 2024 https://doi.org/10.1287/stsc.2024.0189 Copyright: © 2024 The Author(s) Abstract. Scholars argue that artificial intelligence (AI) can generate genuine novelty and new knowledge and, in turn, that AI and computational models of cognition will replace human decision making under uncertainty. We disagree. We argue that AI’s data-based prediction is different from human theory-based causal logic and reasoning. We highlight problems with the decades-old analogy between computers and minds as input–output devices, using large language models as an example. Human cognition is better conceptu- alized as a form of theory-based causal reasoning rather than AI’s emphasis on information processing and data-based prediction. AI uses a probability-based approach to knowledge and is largely backward looking and imitative, whereas human cognition is forward- looking and capable of generating genuine novelty. We introduce the idea of data–belief asymmetries to highlight the difference between AI and human cognition, using the exam- ple of heavier-than-air flight to illustrate our arguments. Theory-based causal reasoning provides a cognitive mechanism for humans to intervene in the world and to engage in directed experimentation to generate new data. Throughout the article, we discuss the implications of our argument for understanding the origins of novelty, new knowledge, and decision making under uncertainty. Open Access Statement: This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License. You are free to download this work and share with others commercially or noncommercially, but cannot change in any way, and you must attribute this work as “ Strategy Sci- ence Copyright © 2024 The Author(s). https:/ /doi.org/10.1287/stsc.2024.0189, used under a Crea- tive Commons Attribution License: https:/ /creativecommons.org/licenses/by-nd/4.0/.” Keywords : cognition • artificial intelligence • prediction • causal reasoning • decision making • strategy • theory-based view Introduction Artificial intelligence (AI) now matches or outperforms humans in any number of games, standardized tests, and cognitive tasks that involve high-level thinking and strategic reasoning. For example, AI engines can readily beat humans in chess, which, for decades, served as a key benchmark of AI capability (Simon 1985a, Bory 2019). AI systems also now perform extremely well in complex board games that involve sophisticated negoti- ation, complex interaction with others, alliances, decep- tion, and understanding other players’ intentions (e.g., Ananthaswamy 2022). Current AI models also outper- form more than 90% of humans in various professional qualification exams, such as the bar exam in law and the certified public accountant exam in accounting (Achiam et al. 2023). AI has also made radical strides in medical diagnoses, beating highly trained medical professionals in diagnosing some illnesses (e.g., Zhou et al. 2023). These rapid advances have led some AI scholar to argue that even the most human of traits, such as consciousness, will soon be replicable by machines (e.g., Goyal and Ben- gio 2022, Butlin et al. 2023). In all, AI is rapidly devising algorithms that “think humanly,” “think rationally,” “act humanly,” and “act rationally” (Csaszar and Stein- berger 2022, pp. 2–3). Given the astonishing progress of AI, Daniel Kahne- man (2018, pp. 609–610, emphasis added) asks (and answers) the logical next question: “Will there be any- thing that is reserved for human beings? Frankly, I don’t see any reason to set limits on what AI can do ... And so it’s very difficult to imagine that with sufficient data there will remain things that only humans can do ... You should replace humans by algorithms when- ever possible.” Kahneman is not alone in this assessment. Davenport and Kirby (2016, p. 29) argue that “we already know that analytics and algorithms are better at creating insights from data than most humans” and “this human/machine performance gap will only increase.” Many scholars claim that AI is likely to outperform 1 STRATEGY SCIENCE Articles in Advance , pp. 1–26 ISSN 2333-2050 (print), ISSN 2333-2077 (online) https://pubsonline.informs.org/journal/stsc Downloaded from informs.org by [2601:680:cd01:ec10:59ae:bab6:e677:2be9] on 04 December 2024, at 13:41 . For personal use only, all rights reserved. humans in most—if not all—forms of reasoning and decision making (e.g., Legg and Hutter 2007, Morris et al. 2023, Grace et al. 2024). Some argue that strategic deci- sion making might also be taken over by AI (Csaszar et al. 2024) or even that science itself will be automated by “AI scientists” (e.g., Lu et al. 2024, Manning et al. 2024). One of the pioneers of AI, Geoffrey Hinton, argues that large language models (LLMs) already are sentient and intelligent and “digital intelligence” will inevitably surpass human “biological intelligence”—if it has not already done so (see Hinton 2023; also see Bengio et al. 2023). Compared with machines, the cognitive and compu- tational limitations of humans are obvious. Humans are biased (Kahneman 2011, Chater et al. 2018). Humans are selective about what data they attend to and sample, and they are susceptible to confirmation bias, motivated reasoning, and hundreds of other cognitive biases (nearly 200 as of last count). In short, humans are bound- edly rational—significantly hampered by their ability to compute and process information (Simon 1955), particu- larly compared with computers (cf. Simon 1990). And the very things that make humans boundedly rational and poor at decision making are seemingly the very things that enable computers to perform well on cogni- tive tasks. The advantage of computers and AI is that they can handle vast amounts of data and process it quickly and in powerful ways. In this paper, we offer a contrarian view of AI relative to human cognition, including its implications for strategy, the emergence of novelty, and decision making under uncertainty. AI builds on the idea that cognition—by both machines and humans—is a generalized form of informa- tion processing: a type of input–output device. To illustrate cognitive differences between humans and computers, we use the example of large language models versus human language learning. We introduce the notion of data–belief (a)symmetry and the role this, respectively, plays in explaining AI and human cognition, using heavier-than- air flight as an extended example. Human cognition is for- ward-looking, necessitating data–belief asymmetries, which are manifest in theories, causal reasoning, and experimentation. We argue that human cognition is driven by forward-looking, theory-based causal logic, which is distinct from the emphasis AI and computational models of cognition place on prediction and backward-looking data. Theory-based causal reasoning enables the gener- ation of new and contrarian data, observations, and experimentation. We highlight the implications of these arguments for understanding the origins of novelty, new knowledge, and decision making under uncertainty. 1 AI 5 Mind: Is Cognition Computation? Modeling the human mind—thinking, rationality, and cognition—has been the central aspiration and ambition behind AI from the 1940s to the present (McCulloch and Pitts 1943, Turing 1948/1992; also see Simon 1955, Hin- ton 1992, McCorduck 2004, Perconti and Plebe 2020). As put by the organizers of the first conference on AI (held at Dartmouth in 1956), their goal was to “proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simu- late it” (McCarthy et al. 2007, p. 12). The commonalities between models of AI and human cognition are not just historical, but these linkages have only deepened in the intervening decades (for a review, see Sun 2023; also see Laird et al. 2017). Computation also underlies many other models of cognition, including the concept of men- tal models (Johnson-Laird 1983), the Bayesian brain, and predictive coding or processing (e.g., Friston and Kiebel 2009; Hohwy 2013, 2020). In fact, cognitive scientist Johnson-Laird (1983, p. 477) goes so far as to argue that “any scientific theory of the mind has to treat it as an automaton.” AI sees cognition as a general form of computation, specifically in which “human thinking is wholly information-processing activity” (Feigenbaum 1963, p. 249; also see Simon 1980). This logic is also captured by computational neuroscientist David Marr (1982, p. 4), who states that “most of the phenomena that are central to us as human beings—the mysteries of life and evolu- tion, of perception and feeling and thought—are primar- ily phenomena of information processing” (cf. Hinton 2023). Both mind and machine are a type of generalized input–output device, in which inputs such as stimuli and cues (data) are processed to yield varied types of outputs, including decisions, capabilities, behaviors, and actions (Simon 1980, 1990; McClelland and Rumel- hart 1981; Hasson et al. 2020). This general model of information processing has been applied to any number of issues and problems at the nexus of AI and cognition, including perception, learning, memory, expertise, search, and decision making (cf. Russell and Norvig 2022). Furthermore, the idea of human mental activity as computation is pervasive in evolutionary arguments. For example, Cosmides and Tooby (2013, pp. 202–203) focus on the “information-processing architecture of the human brain” and further argue that “the brain is a com- puter, that is, a physical system that was designed to process information.” Now, our overall purpose is not to exhaustively review models of AI and cognition, particularly as excel- lent reviews can be found elsewhere (e.g., Goodfellow et al. 2016, Aggarwal 2018, Russell and Norvig 2022). Rather, we simply want to point out the strong emphasis that past, current, and ongoing research places on the similarities between models of AI and human cognition. To assure the reader that we are not creating a caricature of existing work, we have provided relevant, additional detail about AI-cognition similarities in the appendix. Felin and Holweg: AI, Human Cognition, and Causal Reasoning 2 Strategy Science, Articles in Advance , pp. 1–26, © 2024 The Author(s) Downloaded from informs.org by [2601:680:cd01:ec10:59ae:bab6:e677:2be9] on 04 December 2024, at 13:41 . For personal use only, all rights reserved. There, we more exhaustively point out examples of how scholars—from the 1950s to the present—have sought to create an equivalence between AI, machines, and human cognition. In all of this work, cognition and com- putation (and AI) are seen as deeply connected: the underlying premise of this work is that machines and humans are a form of input–output device, in which the same underlying mechanisms of information processing and learning are at play. The focus on computation and information processing also is the axiomatic basis for the concept of bounded rationality (for a review, see Felin et al. 2017). Bounded rationality focuses on human “computational capacities” and their limits (Simon 1955, p. 99), and this idea has deeply shaped fields such as eco- nomics, decision theory, strategy, and the cognitive sciences (e.g., Chater et al. 2018, Kahneman 2003, Pura- nam et al. 2015, Gigerenzer and Goldstein 2024). We disagree with the idea that AI and human cogni- tion share significant similarities as forms of computa- tion for reasons to be discussed next. That said, our aim in making this claim is not to take away from the exciting breakthroughs in AI. Rather, we highlight how the anal- ogy between AI and humans quickly breaks down when it comes to understanding the mind and cognition with important derivative consequences for how we think about the emergence of novelty, new knowledge, and decision making under uncertainty. In the next sec- tion, we delve into a specific example, namely, language learning by machines versus humans, to enable us to make this point more carefully. Machine vs. Human Learning: Different Inputs, Different Outputs While the input–output model of minds and machines— whether we are talking about symbolic or subsymbolic approaches (see the appendix for further detail)—has been a central emphasis of AI and cognitive science, next we highlight some important differences between machine learning and human learning. An apt context for highlighting these differences is to focus on language. Language arguably is “the most defining trait of human cognition (language and its relation with thought),” and therefore, it “can be a true ‘window into the mind’” (Chomsky and Gallega 2020, p. 321; also see Pinker 1994). 2 Language provides an important test and context for understanding human and artificial intelligence. Fur- thermore, some have already argued that large language models are sentient with a few even arguing that they already closely mirror or exceed human cognition (e.g., Binz and Schulz 2023, Hinton 2023)—an assumption that we challenge. At the most basic level, to study any system and its behavior, we need to understand its inputs and outputs. Turing (1948/1992) argued that any form of intelligence, whether human or machine, can be studied as an input–output system. In discussing the possibilities of artificial intelligence—or “intelligent machinery” as he called it—Turing (1950, p. 456, emphasis added) made the analogy to an “untrained infant brain,” saying an infant brain is largely a blank slate, “something like a notebook” with “ little mechanism, and lots of blank sheets” (cf. Turing 1948/1992). According to Turing, these blank sheets are (or need to be) filled with inputs via the process of training and education. Through the early course of its life, an infant or child is taught and receives inputs in the form of language and spoken words that it hears and encounters. Education and train- ing represent the inputs that eventually account for human linguistic capacities and outputs. And in the same way, Turing (1948/1992, p. 107) argues, one can think of an “analogous teaching process applied to machines,” where machines learn from their inputs. Turing lists various settings in which a thinking machine might show that it has learned—including games such as chess or poker, cryptography, or mathematics—and he argues that the “learning of languages would be the most impressive, since it is the most human of these activities” (Turing 1948/1992, p. 117). As human and machine learning are often seen as a similar process, we next focus on key differences using language learning as our example. We then highlight the implications of these differences in learning for decision making and knowl- edge generation both in scientific and economic contexts. How Machines Learn Language. To illustrate the pro- cess of machine learning, next we carefully consider modern LLMs and how they learn. LLMs offer a useful instantiation of machine learning. Learning is essentially generated from scratch—bottom up, directly from the data—through the introduction of vast amounts of train- ing data and the algorithmic processing of the statistical associations and interactions among that data. In the context of an LLM, the training data are composed of enormous amounts of words and text, pulled together from various public sources and the internet. To appreci- ate just how much data and training these models incor- porate, the latest LLMs (as of early 2024) are estimated to include some 13 trillion tokens (a token being the rough equivalent of a word). To put this into context, if a human tried to read this text—say at a speed of 9,000 words/hour (150 words/minute)—it would take more than 164,000 years to read the 13 trillion words of a train- ing data set. The vast corpus of text used to train an LLM is toke- nized to enable natural language processing. This typi- cally involves converting words (or subword units or characters) into numerical sequences or vectors. To illustrate, a sentence such as “The cat sat on the mat” might be tokenized into a sequence such as [10, 123, 56, 21, 90, 78]. Each token is passed through an embedding layer, which converts the token into a dense vector Felin and Holweg: AI, Human Cognition, and Causal Reasoning Strategy Science, Articles in Advance , pp. 1–26, © 2024 The Author(s) 3 Downloaded from informs.org by [2601:680:cd01:ec10:59ae:bab6:e677:2be9] on 04 December 2024, at 13:41 . For personal use only, all rights reserved. representation that captures semantic information, such as its frequency and positional embedding. The embed- ding layer has its own set of parameters (weights) that are learned during training. The attention mechanism introduced with the transformer architecture (Vaswani et al. 2017), touched on by us previously, allows the model to consider each token in the context of all other surrounding tokens and thus to gain an understanding of the wider context. Deep artificial neural networks have turned out to be extremely general and applicable not just to text, but to varied domains such as image rec- ognition and computer vision, including multimodal applications that combine various types of data (for exam- ple, enabling the creation of images from text prompts). 3 From the vast data that serves as its training input, the LLM learns associations and correlations between vari- ous statistical and distributional elements of language: specific words relative to each other, their relationships, ordering, frequencies, and so forth. These statistical associations are based on the patterns of word usage, context, syntax, and semantics found within the training data set. The model develops an understanding of how words and phrases tend to co-occur in varied contexts. The model does not just learn associations, but also understands correlations between different linguistic elements. In other words, it discerns that certain words are more likely to appear in specific contexts. Now, whereas the above is not a technical introduc- tion to LLMs, it offers the broad outlines of the process to the degree that it is relevant for our argument (for a detailed review, see Naveed et al. 2023, Chang et al. 2024, Minaee et al. 2024; also see Resnik 2024). The end result of this training is an AI model that is capable of language: more specifically, the model is capable of gen- erating fluent and coherent text by using a stochastic approach of next word prediction in response to a prompt. In short, LLM outputs are based on conditional probabilities given the structure of the inputs they have encountered in their training data. Based on this broad outline of how an LLM is trained, we compare this to how humans learn language. We should reiterate, as discussed at the outset of this article, that the basic premise behind models of AI is that there is a symmetry between how machines and humans learn. We think it is important to carefully point out dif- ferences as these provide the foundation for our subse- quent arguments about cognition and the emergence of novelty. How Humans Learn Language Compared with Machines. The differences between human and machine learning— when it comes to language (as well as other domains)— are stark. Whereas LLMs are introduced to and trained with trillions of words of text, human language training happens at a much slower rate. To illustrate, a human infant or child hears—from parents, teachers, siblings, friends, and their surroundings—an average of roughly 20,000 words a day (e.g., Hart and Risley 2003, Gilker- son et al. 2017). So, in its first five years, a child might be exposed to—or trained with—some 36.5 million words. By comparison, LLMs are trained with trillions of tokens within a short time interval of weeks or months. 4 The inputs differ radically in terms of quantity (sheer amount) and also in terms of their quality. 5 Namely, the spoken language to which an infant or young child is (largely) exposed is different from the written language on which an LLM is trained. Spoken language differs significantly from written language in terms of its nature, structure, and purpose. Here, the research on the differences between spoken and written language is highly instructive (e.g., Biber 1991). Spoken language is spontaneous (not meaningfully edited), informal, repeti- tive, and often ephemeral. Written language—on the other hand—is visual and permanent, more carefully crafted, planned, and edited. It is also denser, featuring more complex vocabulary (e.g., Halliday 1989, Tannen 2007). Importantly, the functional purposes and uses of spoken versus written language also differ significantly. Spoken language is immediate, interactive, focused on coordinating, expressing, and practically doing things. Whereas written language also serves these purposes, the emphasis is more on the communication of complex information. The vast bulk of the training data of the LLM is not conversational (for models trained on spoken language or raw audio, see Lakhotia et al. 2021). Rather, written language is more carefully thought out. An LLM is likely to be trained with the works of Shakespeare and Plato, academic publications, public domain books (e.g., from Project Gutenberg), lyrics, blog posts, news articles, and various material from the internet. These data are far cleaner, far more correct grammatically, and orga- nized. Arguably the inputs received by an LLM—in the form of written, edited, and published text—are linguis- tically far superior. In a statistical sense, LLM training data contain less noise and thus offer greater predictive power. Even the vast stores of Wikipedia articles that are included in most LLM training data sets are the end result of thousands of edits to ensure readability, accu- racy, and flow. Clearly, humans learn language under different con- ditions and via different types of inputs. In short, it can readily be argued that the human capacity for language develops differently from how machines learn language in both quantity and quality. Humans (somehow) learn language from extremely sparse, impoverished, and highly unsystematic inputs and data (Chomsky 1975). Compared with LLMs, human linguistic capabilities are radically “underdetermined” by the inputs. That is, the relatively sparse linguistic inputs can scarcely account for the radically novel outputs generated by humans. 6 Beyond the quantitative and qualitative differences in inputs (when it comes language learning by LLMs Felin and Holweg: AI, Human Cognition, and Causal Reasoning 4 Strategy Science, Articles in Advance , pp. 1–26, © 2024 The Author(s) Downloaded from informs.org by [2601:680:cd01:ec10:59ae:bab6:e677:2be9] on 04 December 2024, at 13:41 . For personal use only, all rights reserved. versus humans), it is important to compare the linguistic outputs and capabilities of machines versus humans. In terms of output, LLMs are said to be generative (the acronym GPT stands for generative pretrained transformer). But in what sense are LLMs generative? They are gen- erative in the specific sense that they are able to create novel outputs by probabilistically sampling from the vast combinatorial possibilities in the associational and correlational network of word frequencies, positional encodings, and co-occurrences encountered in the train- ing data (Vaswani et al. 2017). 7 The LLM is generative in the sense that the text that is produced is not simply pla- giarized or copied verbatim from existing sources con- tained in the pretraining data (McCoy et al. 2023). In the process of generating text, the parameters (weights and biases) determine how much influence different parts of the training data probabilistically have on the output. For example, in a sentence-completion task, the weights—developed from the corpus of the training data—help the model decide which words are most likely to come next based on the context provided by the input. The output is statistically derived (or, put differ- ently, probabilistically drawn) from the training data’s underlying linguistic structure. The outputs, therefore, have compositional novelty (in terms of novel ways of saying the same thing—more on this below), and they also manifest some analogical generalization (McCoy et al. 2023). That said, any assessment of how good an LLM is needs to recognize “the problem that [LLMs] were trained to solve: next-word prediction” (McCoy et al. 2024, p. 4). And as next-word prediction engines, LLMs certainly demonstrate exceptional capabilities. Beyond Mirroring: Can AI Generate Genuine Novelty? So far we have summarized the central elements of a particular AI system—an LLM—and compared it with humans. Next, we further address whether an AI can be said to be intelligent and whether it can generate genu- ine novelty and new knowledge. Whereas our focus remains on LLMs, we extend our arguments to other forms of AI and cognitive approaches that focus on data and prediction. We concurrently raise questions about whether an AI system meaningfully can originate new knowledge and engage in decision making under uncertainty. AI: Intelligence and New Knowledge? As we foreshadow above, an AI such as an LLM seems to mirror the inputs with which it has been trained rather than meaningfully manifest some form of intelligence. But beyond next-word prediction and linguistic fluency, could an LLM do a better job than humans in decision making under uncertainty (e.g., Csaszar et al. 2024; cf. Kahneman 2018), or could an LLM or AI scientist per- haps even automate science itself (e.g., Lu et al. 2024, Manning et al. 2024; also see Kıcıman et al. 2023; Agra- wal et al. 2023, 2024)? 8 Without question, LLMs seem to manifest sparks of intelligence. But intelligence is not simply memorization or the ability to restate or paraphrase information in var- ious ways. We argue that LLMs appear intelligent because they capitalize on the fact that the same thing can be stated, said, and represented in indefinite ways. This is readily illustrated by the fact that the revolution- ary breakthrough that gave rise to LLMs—the trans- former architecture—was developed in the context of language translation (Vaswani et al. 2017). In an impor- tant sense, LLMs can be seen as translation generalized. They represent a generalized technology for translating one way of saying things into another way of saying the same thing. Translation, after all, is an effort to represent and accurately mirror something in a different way—to represent the same thing in a different language or with a different set of words or more abstractly: to represent the same thing in a different format. LLMs serve this rep- resentational and mirroring function remarkably well. This representational and mirroring function from lan- guage to language is generalized to a process that takes one way of saying something and generates another way of saying the same thing. Stochastic next-word pre- diction using conditional probabilities—based on the weights and parameters derived from vast training data sets—allows for surprisingly rich combinatorial out- puts. The learning of the LLM is embodied in the rela- tionships found between words that are sampled to enable stochastic generativity, in which the outputs mir- ror past inputs. With vast data, an LLM is good at prob- abilistically and fluently predicting the next word. But, as we discuss, the fluency with which LLMs seem to pre- dict and generate outputs dupes us into seeing them as intelligent, as if they are engaging in far more than mere mirroring or translation. Before revisiting our question of whether an AI such as an LLM could actually originate novelty or engage in some form of forward-looking decision making, it is worth highlighting metaphorical similarities between AI and cognitive architectures based on prediction. For example, consider a cognitive approach such as predic- tive processing (Pezzulo et al. 2024), which shares broad similarities with active inference, the free energy princi- ple, the Bayesian brain, and predictive coding. At a high level, both LLMs and predictive processing seek to engage in a similar process, namely, error minimization and iterative optimization, in which the systems are essentially navigating a high-dimensional space to find a state that minimizes both error and surprise. LLMs learn from the training data, and predictive processing learns from its environment (cf. Hohwy 2020). LLMs aim to reduce the difference between their probabilistic Felin and Holweg: AI, Human Cognition, and Causal Reasoning Strategy Science, Articles in Advance , pp. 1–26, © 2024 The Author(s) 5 Downloaded from informs.org by [2601:680:cd01:ec10:59ae:bab6:e677:2be9] on 04 December 2024, at 13:41 . For personal use only, all rights reserved. predictions (the next word in a sentence) and the actual outcomes (the real next word), thereby improving their accuracy. Predictive processing, as a cognitive theory, posits that the brain continuously predicts sensory input and minimizes the error between its predictions and actual sensory input. The capability of each to predict— whether a word or a perception—is a function of past inputs. Large language models seek to predict the most likely next word based on training data, and active infer- ence seeks to predict the most likely next percept or action. Both approaches are wildly conservative (tied to past data) as they seek to reduce surprise—or to engage in prediction as error minimization (Hohwy 2013). 9 Back-propagation, a fundamental mechanism in train- ing neural networks, and the concept of error minimiza- tion in predictive processing (Friston and Kiebel 2009) share a broad conceptual similarity in that both involve iterative adjustments to minimize some form of error or discrepancy. Both generate a prediction based on past inputs. Both back-propagation and error minimization in predictive processing involve adjusting an internal model (neural network weights in AI and hierarchical brain models in neuroscience) to reduce error (or, in machine learning terms, minimize the loss function). With this architecture—focused on error minimiza- tion and surprise reduction—can an LLM or any prediction-oriented, cognitive AI truly generate some form of new knowledge? Beyond memorizing, translat- ing, restating, or mirroring the text with which it has been trained, can an LLM generate new knowledge? We do not believe LLMs or input–output–based cog- nitive systems can do this—at least not beyond random flukes that might emerge because of their stochastic nature. 10 There is no forward-looking mechanism or unique causal logic built into these systems. It is impor- tant to clearly delineate why this is the case as some argue and anticipate that LLMs will replace human deci- sion makers in uncertain contexts such as strategy and even science itself. For example, Csaszar et al. (2024) argue that “the corpora used to train LLMs include valuable information for SDM, such as consumer prefer- ences, competitor information, and strategy knowledge” and point to how an AI can use various decision-making tools to generate business plans and strategy (Csaszar et al. 2024, p. 2). And Manning et al. (2024) even argue that LLMs will automate social science given their seeming ability to generate hypotheses and causal models, includ- ing testing them (also see Lu et al. 2024). These claims are vastly overstated. One way to think about this is that a prediction-oriented AI such as an LLM can essentially be seen as possessing Wikipedia- level knowledge. On any number of topics (if contained in the training data), an LLM can summarize, represent, and mirror the words it has encountered in various dif- ferent and new ways. On any given topic—again, if suf- ficiently represented in the training data—an LLM can generate indefinite numbers of coherent, fluent, and well-written Wikipedia articles by drawing on the con- ditional probabilities it has learned. But, just as a subject- matter expert is unlikely to learn anything new about the expert’s specialty from a Wikipedia article within the expert’s domain of expertise, an LLM is unlikely to somehow bootstrap knowledge beyond the combinato- rial possibilities of the word associations it has encoun- tered in the past. It has no forward-looking mechanism for doing so. There is also good evidence to suggest that when an LLM encounters (is prompted with) a reasoning task, it merely reproduces the linguistic answers (about reason- ing) it has encountered in the training data rather than engaging in any form of actual, on-the-fly reasoning. If the wording of a reasoning task—such as the Wason selection task or the Monty Hall problem—is changed only slightly, LLM performance declines significantly below human performance, and the mistakes of the LLM are glaringly obvious to humans (e.g., Hong et al. 2024). LLMs are not meaningfully engaged in any form of real-time reasoning (as assumed by Lu et al. 2024, Manning et al. 2024). Rather, they are merely repeating the word structures associated with reasoning, which they have encountered in the training data. This effect can also be shown empirically as training LLMs on their past output leads to a rapid decline in performance and even their collapse (Shumailov et al. 2024). Importantly, LLMs memorize and regurgitate the words associated with reasoning but do not engage in on-the-fly reason- ing of any sort. 11 This is why Francois Chollet (2019) created the “abstraction and reasoning corpus” as a chal- lenge or test to see if an AI system can actually solve new problems (that is, problems it has not encountered in its training data) without merely resorting to memorized answers and solutions encountered in the past (which captures the present state of AI systems, including LLMs). 12 That said, our goal is not to dismiss the remarkable feats of LLMs or other forms of AI or applications of machine learning. The fact that an LLM can outperform most humans in varied types of tests and exams is remarkable (Achiam et al. 2023). But this is because it has encountered this information, memorized it, and is able to repeat it in fluent ways. An LLM essentially has a superhuman capacity for memorization and an ability to summarize memorized word structures in diverse ways. In all, certainly the idea of LLMs as “stochastic parrots” or “glorified auto-complete” (Bender et al. 2021) underestimates their ability. But, equally, ascrib- ing LLMs the ability to actually reason and generate new knowledge vastly overestimates their ability. LLMs are essentially powerful and creative imitation engines in stochastically and probabilistically assembling words though not linguistically innovative compared with children (see Yiu et al. 2023). The idea that LLMs Felin and Holweg: AI, Human Cognition, and Causal Reasoning 6 Strategy Science, Articles in Advance , pp. 1–26, © 2024 The Author(s) Downloaded from informs.org by [2601:680:cd01:ec10:59ae:bab6:e677:2be9] on 04 December 2024, at 13:41 . For personal use only, all rights reserved. somehow generate new-to-the-world knowledge—or feature something like human consciousness—seems to be a significant stretch (though, see Butlin et al. 2023, Hinton 2023). In sum, the generativity of these models is a type of lowercase “g” generativity that shows up in the form of the unique sentences that creatively summarize and repackage existing knowledge. To illustrate the problem of generating something novel—such as new knowledge—with an LLM, imagine the following thought experiment. Imagine an LLM in the year 1633, where the LLM’s training data incorpo- rates all the scientific and other texts published by humans to that point in history. If the LLM were asked about Galileo’s heliocentric view, how would it respond? Because the LLM would probabilistically sample from the association and correlation-based word structure of its vast training data—again, everything that has so far been written (including all the scientific writings about the structure of the cosmos)—it would only restate, rep- resent, and mirror the accumulated scientific consensus. The training data set for the LLM would overwhel- mingly feature texts with word structures supporting a geocentric view in the form of the work of Aristotle, Ptol- emy, and many others. Ptolemy’s careful trigonometric and geometric calculations, along with his astronomic observations, would be included in support of a geocen- tric view as represented in the many texts that would have summarized the geocentric view (such as de Sacro- bosco’s popular textbook De saphera mundi ). These texts would feature word associations that highlight how the motions and movements of the planets could be pre- dicted with remarkable accuracy with the predominant geocentric view. The evidence—as inferred from the repeated word associations found in the training data— would overwhelmingly be against Galileo. LLMs do not have any way of accessing truth (for example, through experimentation or counterfactuals) beyond mirroring and restating what is found in the text. Even if alternative or heretical views were included in the training data (such as the work of Copernicus even though his work was largely banned), the logic of this work would be dwarfed by all the texts and materials that supported the predominant geocentric paradigm. 13 The overwhelming corpus of thousands of years of geo- centric texts would vastly outweigh Galileo’s view or anything supporting it. An LLM’s model of truth or knowledge is solely statistical, relying on frequency and probability. Outputs are influenced by the frequency with which an idea is mentioned in the training data as reflected by associated word structures. For example, the frequency with which the geocentric view has been mentioned, summarized, and discussed in the training data necessarily imprints itself onto the output of the LLM as truth. As the LLM has no actual grounding in truth beyond the statistical relationships between words, it would say that Galileo’s view and belief is delusional and in no way grounded in science. A neural network such as an LLM might, in fact, include any number of delusional beliefs, including beliefs that turned out to eventually be correct (such as Galileo’s) and also beliefs that objectively were (and still are) delusional. Ex ante, there is no way for an LLM to arbitrate between the two. For example, the eminent astronomer Tyco Brahe made and famously published extensive claims about astrology, the idea that celestial bodies and their movement directly impact individual human fates as well as political and other affairs. His astrological writings were popular not just among some scientists, but also among the educated elite. A hypo- thetical LLM (in 1633) would have no way of arbitrating between Galileo’s (seeming) delusions about heliocentr- ism nor Brahe’s (actual) delusions about astrology. Our hypothetical LLM would be far more likely to have claimed that Brahe’s astrological claims are true than that Galileo’s argument about heliocentrism is true. The LLM can only represent and mirror the predominant and existing conceptions—in this case, the support for the geocentric view of the univer