Can’t help processing numbers with text: Eye-tracking evidence for simultaneous instead of sequential processing of text and numbers in arithmetic word problems

Solving word problems requires solvers to create a correct mental representation (sometimes also referred to as ‘mental model’) of the word problems. To our understanding, a mental representation is a mental framework that individuals build during problem-solving. It combines all key information—like details in the text, the situation, potentially required mathematical operations, and involved numbers—in a single, organized structure. Mental representations are dynamic and can be adjusted when more information is understood. They allow problem solvers to find solutions. We consider number processing to be part of building the mental representation, rather than being run only once the mental representation already exists.

Naturally, properties of the numbers used in word problems play an important role in word problem solving. Numerical properties such as parity and magnitude or the type of operations and numbers have rarely been thoroughly investigated independently from one another in word problems, despite their important role in current numerical cognition research (see Daroczy et al., 2015, for a review). Also, perceiving and semantically understanding multi-digit numbers is more challenging than single-digit numbers (Nuerk et al., 2011, 2015), and arithmetic complexity typically increases with the number of digits in standard arithmetic tasks. This phenomenon appears to hold true for word problems as well. For instance, Thevenot and Oakhill (2005) compared the influence of 3- and 2-digit numbers on word problem solution strategies and demonstrated that calculating more complex 3-digit numbers facilitated alternative strategies among the participants. However, the authors suggested that this facilitation was due to higher workload and working memory demands rather than due to the timing of number processing. Similar results were observed by Brissiaud and Sander (2010), who manipulated the size and order of the numbers and operations and found that number magnitude and order determined which strategy was most likely to be used.

However, not only the number of digits is relevant in a word problem. For example, children perform better if the numbers involve multiples of 25 (Koedinger & Nathan, 2004). Even when the number of digits is controlled, the complexity of place-value processing can influence word problem performance. Moreover, the carry and borrow effects play a role in word problems (e.g., Dresen et al., 2020). Carrying (in addition) occurs when the sum of digits in a column exceeds the base value (i.e., 10 in base-10 system), transferring the excess to the adjacent higher place value. Borrowing (in subtraction) occurs when a digit in the minuend is less than the corresponding digit in the subtrahend, necessitating the transfer of a unit from the next higher place value to proceed with the subtraction. RT have been found to be longer, and ER higher, in carry or borrow compared to non-carry or non-borrow addition and subtraction arithmetic problems (carry effect: Ashcraft, 1995; Moeller et al., 2011b; borrow effect: Artemenko et al., 2018). Similar carry and borrow effects have also been observed in word problems (e.g., Daroczy et al., 2020).

Apart from the number of digits and the necessity of carrying and borrowing, the problem size can also influence performance in word problems (De Corte, 1988) The larger the problem size, the slower the responses and the more errors are made. Furthermore, the operation to be carried out plays an important role in word problem solving, such that subtraction problems are more difficult than addition problems (e.g., Daroczy et al., 2020). Importantly, this is connected to the semantics of the word problems, so operations cannot be seen as purely numerical factors.

For the present work, number processing and text processing are used as rather broad terms. Number processing involves basic, automatic steps, such as recognizing numerical information and understanding the numbers’ magnitudes. Additionally, number processing includes integrating numbers into the mental representation, and potentially also selecting operations and performing calculations. Note that in the literature, the term number processing also sometimes refers to the mere understanding of the semantic meaning of numbers in terms of their magnitude, which has been shown to take place automatically even when not intended or required by the task (Tzelgov et al., 1992). In the current study, however, we expand this restrictive definition of number processing.

Taken together, numerical complexity in arithmetic word problems (e.g., number of digits in used numbers, necessity of a carry or borrow operation, problem size) plays an important role for performance. Although some studies acknowledge numerical effects in word problems (Daroczy et al., 2015, for an overview), there is uncertainty about whether number processing already happens in parallel to text processing, or whether number processing takes place only afterwards.

Several types of models have been proposed to explain cognitive processing in arithmetic word problems. While some models propose sequential processing with different phases following one another, others propose interactive processing with different kinds of information being processed in parallel.

The first models explaining arithmetic word problem solving came up in 1984 and 1985 and were clearly sequential models. Mayer et al. (1984) described different phases that play a role in solving word problems. They stated that after the initial reading, the text is mentally represented, and this mental representation is then used to initiate the specific problem-solving process. One year later, Kintsch and Greeno (1985) published the Propositional Theory, which suggests that the mental representation is created based on a so-called propositional text base. According to this theory, which falls into the category of schema models, processing steps in problem solving are sequential. Schemas are stored in long-term memory, activated after the text processing, and then filled out with numbers (Kintsch & Greeno, 1985). In some types of word problems, using schemas as a strategy may be effective (Marshall, 1995), while in other cases, it may not work, and the situational context of a word problem might hinder the application of schemas (e.g., Stern & Lehrndorfer, 1992). Therefore, schema models has been challenged across many studies by situation models (e.g., Thevenot et al., 2007). Situation models suggest that an episodic situation model is created from the word problem, which is then translated into a mental representation. Currently, there is a consensus that a mental representation of the situation is indeed required (e.g., Pongsakdi et al., 2020). Ultimately, all these models can be seen as sequential models.

In a different theory, Bergqvist and Österholm (2010) proposed that reading and solving task demands are tightly integrated. The authors argue for a magnitude-based, more dynamic mental representation. Dynamic mental representations of word problems would be in line with conclusions drawn from studies on text processing (in terms of reading comprehension in non-mathematical stimuli), where mental representations have been claimed to be updated regularly and where inferences have been shown to be made on-line while reading (Gernsbacher et al., 1992, 1998). In contrast to sequential models by Mayer and colleagues (1984) and by Kintsch and Greeno (1985), the model by Bergqvist and Österholm is clearly interactive, with different parts of cognitive processing happening simultaneously.

A more recent model, which can be understood as an extension of situation models, is Gros et al.’s (2020) semantic congruence model (SECO). It identifies three factors crucial for solving word problems: mathematical semantics (understanding math concepts), algorithms (strategic problem-solving), and world semantics (real-world knowledge). The SECO model also integrates knowledge about the semantic relations between the problem elements and the mental representations of the problems (e.g., Bassok et al., 1998). The nature of mental representations and the influence of semantic congruence have recently been demonstrated by a thoughtfully designed study, where participants were not only asked to solve, but also to recall the word problems in an unexpected memory task (Gros et al., 2024). Importantly, in the SECO model, mathematical knowledge seems connected to language rather than to number processing such as addition and subtraction. The SECO model also extends previous models with a feedback loop. Some models, especially from educational research (e.g., Blum & Leiss, 2005), also use a feedback loop. However, in this case, the existing mental representation is renewed after an incorrect strategy, and these models do not sufficiently explain what happens when only the numerical difficulty is altered, without any connection to the semantics of the text. The SECO model does not make a clear statement about whether cognitive processing of the three postulated factors (i.e., mathematical semantics, algorithms, and world semantics) takes place sequentially or in parallel. However, the SECO model includes a possible recoding loop which relies on mathematical semantics. This recoding loop can be activated in the case of semantically incongruent information to correct the original mental representation, making the model non-sequential.

An analogy to the relation between text and number processing in mathematical word problems is the relation between syntax and semantics in language. On the one hand, language establishes the structure of the problem, and on the other hand, it allows inferences about the numerical problem (for an overview see Singer, 1994). Such inferences are not necessarily logical, but can also be pragmatic (Harris & Monaco, 1978; Singer, 1988). In fact, in non-standard word problems, most inferences are not strictly logical, but pragmatic in the sense that they rely on people’s common knowledge about the world. For instance, a popular non-standard word problem is: “Ten birds are sitting on a tree. The hunter shoots down two birds. How many birds are still on the tree?”. The logical inference (without using world knowledge) would be that 8 birds are left, because only 2 of the 10 birds are shot. The pragmatic inference, which many children and adults do not use in such word problems, is that when some of the birds are shot, all other birds fly away. In the 1990s and afterward, significant research examined the relationship of syntax and semantics in language (van Valin, 2005). Both play a critical role in word problems. For instance, Daroczy et al. (2020) showed that both syntactic (e.g., nominalization) and semantic features (e.g., lexical consistency of keywords) play a role in understanding and solving word problems. Hence, in a similar way the interaction of syntax and semantics was studied before, we now investigated the interplay between text processing and number processing in word problems. We believe that text processing and number processing are as deeply intertwined as syntax and semantics in text and discourse processing.

To sum up, some models about cognitive processing in arithmetic word problems are sequential and assume that number processing happens only after text processing. According to other models, number processing may occur in parallel with text processing. Moreover, some models are agnostic about when number processing takes place when solving arithmetic word problems. The aim of the current study was to determine whether number processing occurs at an early stage, thereby contrasting sequential and interactive models of word problem-solving.

After this overview of the models explaining arithmetic word-problem processing, we would like to revisit them in terms of how they address number processing within arithmetic word-problems. Different models interpret and utilize numbers in distinct ways, giving them different roles and importance. However, most models assign limited importance to numbers. Instead, most studies identify semantic and linguistic factors, rather than numerical ones, as the reasons why people fail to construct a correct mental representation of the problem. The review by Jaffe and Bolger (2023) argues that cognitive load increases when numbers are presented as text rather than in Arabic format; the reason for this is that number processing cannot circumvent a phonological step and be directly mapped onto numerical values (Damian, 2004). This suggests that number processing might interact with other problem-solving steps, but the review does not address the possibility of a magnitude-based mental representation.

In contrast, the schema model (e.g., Kintsch & Greeno, 1985) is rather sequential. Here, the emphasis lies on first activating schemas in long-term memory and afterwards instantiating them with specific numerical values. Hence, numbers only play a role after the conceptualization, indicating a more significant role of linguistic features and reading. In line with this, Fuchs et al. (2018) describe text processing and number processing in word problems as two successive phases. Specifically, in the first phase, the text’s essential ideas are captured to build up an adequate mental representation of the problem. This includes a semantic understanding of the words and potentially also of the numbers (i.e., the extraction of their magnitude, which is highly automatic; e.g., see Tzelgov et al., 1992). In the second phase, the solution-relevant mathematical operations are deduced from the representation and calculations are performed by executing the chosen operations. According to this, numbers seem to play a role in the later stages, aligning with the last step of problem-solving.

The SECO model (Gros et al., 2020) partly incorporates the role of numbers in one of its essential components, namely mathematical semantics. Mathematical semantics is defined as the solver’s mathematical knowledge needed for the given problem statement, which enables the identification of solving algorithms. The SECO model therefore indirectly addresses the challenges associated with numbers in arithmetic word problems, but it does not explain why problem-solving performance changes when the numbers are changed (e.g., in the case of carry/borrow), particularly when the numbers are altered in a way that does not affect the underlying mathematical semantics.

Across these models, a common step is the translation of the interpreted structure into a mathematical operation, which is assumed to occur at a later stage. Interestingly, the choice of words in the text is claimed to have a significant influence on this translation. However, the models do not adequately address the impact of choice of numbers.

In contrast, Bergqvist and Österholm (2010) argue that number processing is highly integrated and cannot be separated from other problem-solving processes such as text processing. According to their model, heuristic approaches allow number processing to start during the initial reading and argue for a magnitude-based mental representation, however their assumption was not empirically proven. There are indeed very few empirical studies investigating the existence of magnitude-based mental representations. For instance, Orrantia and Múñez’ (2013) study provided evidence that an analog magnitude-based mental representation is consistently activated during word-problem solving. In their study, participants solved word problems and then completed a figure discrimination task. Participants demonstrated slower discrimination when the magnitude relationship between figures did not align with the implied quantities' relationship in the sentence they read. However, a potential criticism emerges concerning whether participants formed a genuine representation of magnitude: The figures used in the discrimination task only reflected the relationship between variables in the relational sentence (e.g., John > Peter), failing to capture relative magnitudes based on the numbers presented in the word problem. It is crucial to note that these findings specifically pertain to compare problems and do not extend to change problems. Compare word problems involve evaluating the differences or similarities between two or more items, quantities, or scenarios to understand their relationship better. Change word problems deal with understanding how a quantity increases or decreases over time or through a series of events, focusing on the transformation from one state to another. Additionally, the measurements were taken post problem-solving rather than during the problem-solving process itself, which is an issue we aim to address.

It remains unclear whether aspects determining the numerical complexity of word problems that we discussed above speak for sequential or interactive models of word problems. The question arises whether more demanding and error-prone calculations only manifest their effect on performance in later processing stages (i.e., in line with models favoring sequential processing phases), or whether they already influence performance at an earlier stage (i.e., in line with models favoring interactive processing phases). As stated above, one methodological problem is that many studies rely only on conventional outcome variables such as RT and ER, which allow to understand the final phase of the word-problem solving process, but do not allow us to dive into the process itself. In contrast, eye-tracking can help to reveal mental representations and cognitive processes during the solving process which cannot be detected by analyzing the final phase of behavior and can also not always be reported consciously (De Corte & Verschaffel, 1986).

We hypothesize that number processing begins early and simultaneously with text processing, where the text is read and mentally conceptualized. To test this hypothesis, we used a straightforward concept: In the present study, we tested whether individuals looked more often or for longer durations at numbers in word problems involving more complex calculations compared to word problems with simpler calculations, even when no calculation was necessary. For this purpose, we designed solvable and non-solvable word problems. In the latter ones, participants had to determine that the word problem is non-solvable. Non-solvable word problems fall into the category of non-routine or non-standard word problems. Such problems cannot be solved using known methods or formulas (Artut, 2016), their solutions are not immediately apparent (Celebioglu et al., 2010; Murdiyani, 2018), and success often relies more on logical thinking than on traditional mathematical knowledge and skills acquired in school (e.g., Abdullah et al., 2014). To realize that the word problem lacks a solution, participants do not need to consider the numbers; they aredispensable, at least in the present study. Please note that certain types of non-solvable word problems may require individuals to perform calculations to realize that they are non-solvable. In such cases, number processing is necessary (e.g., “John has 13 apples. Mary has 36 fewer. How many apples does Mary have?”). However, in our study, we focus exclusively on non-solvable word problems where number processing is not needed to determine that the problem is non-solvable. However, participants might engage in the direct translation strategy described by Hegarty et al. (1992, 1995), using the numbers and keywords to perform a calculation. Outcomes from this approach will be excluded from the analysis because the mental representation of a non-solvable word problem must be incorrect if a calculation is carried out. It is possible that individuals begin with the direct translation strategy and later recognize that the problem is non-solvable. They might then alter their mental representation of the word problem or give up. However, even in giving up, there’s a recognition of a mismatch with their initialproblem representation, indicating at least partial integration of text processing (including the creation of a mental representation) and number processing (including potential calculation). Note that even if people realize the problems are non-solvable and create the correct problem representation, we will still not know when and how they use the numbers. Therefore, we manipulated the complexity of the calculation, specifically the necessity of carrying or borrowing, by including problems that contained either the simpler or the more complex hypothetical calculation. Theoretically, if a correct mental representation of the situation is created, and people realize in time that number processing including calculating is unnecessary, they will not use their cognitive resources for calculations. This would only happen if number processing was integrated from the very beginning.

To adequately investigate the cognitive processes during solving arithmetic word problems, the method of eye-tracking has proven to be useful for word problems in numerous studies (e.g., De Corte et al., 1990; Hegarty et al., 1992; Knoblich et al., 2001). Eye-tracking data have been demonstrated to be reliable and valid (e.g., Lai et al., 2013; Susac et al., 2014). Eye-tracking methodology appears to be particularly valuable for studying processes, as demonstrated by Strohmaier et al. (2020). It can help reveal mental representations and assess mathematical cognitive processes that cannot be consciously reported (De Corte & Verschaffel, 1986). The central assumption of eye-movement analysis is the eye-mind assumption, which posits that participants look at the element that is currently being processed (Just & Carpenter, 1980). Consequently, the fixation on words should indicate current text processing, whereas the fixation on numbers should indicate current number processing. Please note that the eye-mind assumption has been challenged several times, and it has been shown that people do not necessarily have to look at a specific area to process information. Nevertheless, Moeller et al. (2011b) found significantly longer attention on carry addition problems compared to non-carry addition problems, suggesting that increased attention on numbers is a valid reflection of deeper processing in a specific area.

However, the question is how these metrics are related to different parts and stages of word problem solving. An important distinction to be made is between static and dynamic eye-tracking variables. People can fixate, jump to, and revisit specific areas in word problems. Lai et al. (2013) and Strohmaier et al. (2019) define fixations as relatively stable states of eye movement and saccades as rapid, straight movements to the next fixated point. Regressions can be understood as all fixations made after the respective area has been left for the first time; they are relatively stable states of eye movement after the first visit. Fixation duration (FD), fixation count (FC), regression duration (RD), and regression count (RC) are the most common measures in word problem solving research (Strohmaier et al., 2019). These four eye-tracking variables are static variables because they describe stable states of the eye, rather than dynamic eye tracking variables, which describe movements from one area to the next. FC and FD reflect the amount of attention invested and are related to processing difficulty (van der Schoot et al., 2009). To obtain the RD or RC, the first-pass FC or FD within the areas of interest (AOI) is subtracted from the total FD or FC (Lai et al., 2013). Regressions to numbers might also reflect number processing, in terms of the execution of the solution-relevant mathematical operations. This is also supported by Dresen et al. (2020) who found significantly longer durations of first fixations and regressions in carry addition problems compared to non-carry addition problems. Therefore, we expect to see similar differences in these measures between carry and non-carry contexts in solvable and even non-solvable word problems.

In previous word problem studies, mostly static eye-tracking measures were applied. Nevertheless, eye movements have a dynamic character, and the eyes move from one fixation to another with rapid movements (i.e., saccades). These movements can be analyzed via scan paths (e.g., Jian & Wu, 2015) or sequence charts (e.g., Strohmaier et al., 2018), for example. Scan paths are sequences of readers’ fixations and saccades. Most of these measures are exploratory in mathematical cognition, and it is challenging to link eye movements to cognitive processes. To clarify the relationship between text and number processing, it is important to examine dynamic shifts occurring between the text and the numbers. Eye movements (i.e., transitions) between text parts, between numbers and text, and between numbers can be seen as indicators for reading comprehension, calculation, and overlapping processes. Movements between text and numbers could suggest that these elements are combined and considered together (Johnson & Mayer, 2012). Regressions to previously read text often signals reading comprehension (e.g., Rayner, 1998; Verschaffel et al., 1992), whereas returning to numbers suggests active engagement in calculation (Hegarty et al., 1995). In eye-tracking research, transitions have been analyzed in various studies, though many focus on graphic reading (e.g., Malone et al., 2020) or fields such as chemistry (Rodemer et al., 2020). However, they have not yet been extensively studied in the context of word problems and mathematical cognition.

In summary, eye-tracking is a valid method for measuring cognitive processes in arithmetic word problems. If the number processing plays a role, attention (fixations, regressions, and transitions) should be drawn to areas of the problems containing numbers.

The goal of the present study is to investigate whether number processing (i.e., recognizing numbers, semantically understanding their magnitude, and potentially performing calculations) in arithmetic word problems occurs simultaneously with text processing or only afterwards. The research specifically examines eye movements, analyzing if people look more often or for longer periods at numbers in more complex problems than in simpler ones, even when a calculation is not needed and participants are aware of this (i.e., build the correct mental representation). If number processing occurs simultaneously with other problem-solving processes such as text processing, there will be an increase in attention (i.e., fixations and regressions on the numbers, and more transitions between numbers or between numbers and text parts) in the non-solvable word problems involving carry or borrow addition or subtraction operations as compared to the non-solvable word problems involving non-carry and non-borrow operations, even when participants categorize them correctly as non-solvable.

Furthermore, we anticipate that the same patterns will emerge in solvable word problems, which require calculations. Like in our pilot study with children (Roth, 2021), we expect lower ER in solvable word problems compared to non-solvable ones, replicating the findings of Jiménez and Verschaffel (2014) and Verschaffel et al. (1994). Additionally, we anticipate longer RT in solvable problems due to the time needed for calculations. Within the category of solvable word problems, we hypothesize that the carry or borrow condition will result in longer RT and higher ER compared to the non-carry or non-borrow condition, as indicated by many studies (e.g., Artemenko et al., 2018; Ashcraft & Stazyk, 1981; Daroczy et al., 2020; Moeller et al., 2011b; Radler et al., 2014). As previous studies have shown higher RT in subtraction compared to addition problems (e.g., Daroczy et al., 2020; Radler et al., 2014), suggesting that subtraction is more challenging than addition, we expect to find higher RT and ER in solvable subtraction compared to addition problems in our study as well. Similarly, regarding the measured eye movements, we hypothesize more and longer fixations and regressions on the numbers (AOI) as well as more transitions between numbers or between numbers and text parts in solvable subtraction than addition problems.

Understanding the sequence of cognitive processes in word problem solving, specifically the role of number processing relative to the formation of a mental representation, is crucial for effective education and cognitive research. If number processing occurs simultaneously within the formation of a mentalrepresentation, educators should not only focus on enhancing both text comprehension and numerical skills, but also be aware that teaching a strictly sequential process does not correspond to how healthy adults with an average performance level solve word problems. Nevertheless, children may rely on a more sequential strategy than adults. As children develop proficiency, they may gradually shift towards more simultaneous text and number processing, becoming more similar to the typical strategy used by adults. An analogous case can be seen in strategies for calculation tasks with distractors: adults show flexibility in choosing correct answers or distractors, whereas children consistently select distractors (Moeller et al., 2011a, 2011b). While this pertains to calculation tasks, similar differences may be conceivable in word problems. Understanding the sequence of cognitive processes in arithmetic word-problem solving may eventually help in tailoring beneficial and avoiding inefficient teaching strategies, enhancing students’ performance.

The sample consisted of 63 adults (M_age = 23.14, SD_age = 5.87, N_female = 42, N_male = 21), who were recruited through a mailing list and a recruitment system at the University of Tübingen as well as through private contacts of the investigators. Inclusion criteria required participants to be at least 18 years old, have German as their native language, and have no neurological disorders. 58 participants were right-handed, and five participants were left-handed. None reported dyslexia or dyscalculia, and 14 participants needed glasses or contact lenses during the experiment. University students in Psychology or Cognitive Science were granted 1.5 h of course credit for their participation.

The study took place in a laboratory at the University of Tübingen, Germany. To avoid daylight fluctuations altered participants’ pupil size during the eye-tracking part, windows were entirely covered. Depending on what worked better in the eye-tracker calibration, the experimenter decided for each participant individually in the very beginning whether to turn on or off the light in the laboratory, and these light conditions were kept stable during the eye-tracking procedure.

In a first step, participants were informed and consented to participate. For this, they were told that their eye movements would be recorded by an eye-tracker, which is an absolutely pain and risk-free method, and that their vocal answers to word problems would be recorded. After giving written consent and filling in a demographic questionnaire (age, gender, handedness, native language, diagnoses of dyslexia or dyscalculia, corrected vision, dominant eye, years of studies, study field or occupation). Participants were asked to sit on a chair in front of a desk in a comfortable position and to move as little as possible while solving the word problems. The microphone and the monitor with the eye-tracker were placed on the desk approximately 20 cm away from their mouth and 50 cm away from their eyes, respectively. To ensure that the participants could continually look at the screen, participants were asked to respond vocally to the word problems. RT in the experiment were measured from the word problem’s appearance on the screen to the vocal response. To ensure the voice key would be triggered and RT would be recorded similarly for all word problems, participants were instructed to start every response with the German word “gleich” (English: “equals”) and to speak loudly, while not making any other noise. To familiarize participants with the setup, to get them used to starting each answer with “gleich”, and to adjust the sensibility of the microphone, participants were asked to answer 11 personal questions (e.g., “Do you have a pet?”), reading out loud 11 numbers, and responding to five solvable practice word problems.

After these practice trials, a target sticker was placed in the center of the participant’s forehead as a reference point for the eye-tracker. We employed monocular eye-tracking for the dominant eye, identified either through participant´s self-report or by assessing eye dominance via a method involving looking through a hand-formed hole and comparing the dominance of the images produced by the left and right eyes. Next, instructions appeared on the screen, in which participants were asked to mentally calculate the solution for the displayed arithmetic word problems as fast and as accurately as possible and say their response out loud while starting with the word “gleich” (English: “equals”). In case of non-solvable word problems, participants were instructed to say “gleich Aufgabe nicht lösbar” (English: “equals problem not solvable”). Participants were asked to look at the fixation point appearing centered in the upper part of the screen after each word problem. Participants were informed that if they would not answer, the next word problem would appear after a while, and that the total word problem test would take around 25 min with a short break halfway through. Next, for the eye-tracker calibration, participants fixated nine points appearing one after the other on the screen. The calibration was validated by repeating the procedure until the deviation between the fixations in a first and second round did not deviate by more than 1.5 degrees for each point, or less than 1.0 degrees averaged over all nine points. The calibration was validated again before each word problem by the participants looking at the fixation point. If the fixation was not recognized correctly, the eye-tracker was recalibrated as in the beginning.

Word problems were presented in white Times New Roman font in 24 pt size on a black background. All four sentences appeared simultaneously centered on the screen with one sentence per line. As soon as the verbal response triggered the voice key or after a maximum of 90 s, the word problem was replaced by a blank black screen. After three seconds, a black and white noise screen appeared for 60 ms, followed by the fixation point. Reponses were both noted down by the experimenter and recorded. The 48 stimuli (i.e., 32 relevant word problems and 16 filler tasks) were presented in two blocks with a break halfway through. The number of solvable vs. non-solvable word problems, of simple vs. complex word problems, of addition vs. subtraction word problems, of word problems with the same story, and of routine vs. non-routine filler tasks were equal in each block. Block order was counterbalanced between participants, and stimulus order was fully randomized within blocks.

In the second part of the study, participants completed questionnaires and tests in a paper–pencil format to assess cognitive covariates that are typically related to the performance in arithmetic word problems. By assessing cognitive characteristics of the sample, potential anomalies are detectable, and the study is more replicable. Moreover, it provided the opportunity to validate correlations between the measured eye-tracking variables and these cognitive covariates. First, figural, verbal, and numerical creativity were assessed by the LO subtest of the BIS-HB (Jäger et al., 2006), the ASK (Schuler and Hell, 2004), and the DR1 subtest of the BIS-HB, respectively. Afterwards, participants’ basic mathematical competencies were assessed by two speed calculation tests (Huber et al., 2013). Finally, verbal and spatial working memory capacity was tested with the letter span task and the Corsi Block-Tapping Test (Corsi, 1972), respectively. In total, participation in the study took between 75 and 90 min. Results for all cognitive covariates assessed in the second part of the study are summarized in the Supplementary Material C.

The word problem part of the study was programmed with the Experiment Builder (SR Research Ltd., version 2.1.4) running on a first computer. Stimuli were presented on a 19-inch monitor with a resolution of 1025 × 768 pixels. Eye movements were recorded via EyeLink 1000 (SR Research Ltd., Mississauga, Canada), an infrared video-based tracking system, using a 500 Hz sampling rate and a 16 mm lens. The eye-tracker was connected to a second computer displaying the eye-tracking software and the participant’s eye movements on a second monitor for the experimenter at any time. Eye-tracking data was stored and later preprocessed for the data analysis in the Data Viewer (SR Research Ltd., version 3.2.1. Vocal responses as well as RT from stimulus presentation to the triggered voice key were recorded with the help of the EMU 0202 USB Audio Interface (Creative Technology Ltd., Singapore). Moreover, a pen and copies of the informed consent as well as the questionnaires and tests for cognitive covariates were needed.

Number and text processing were operationalized by the attention to the numbers and to the text in the word problems, as reflected both by static and dynamic eye-tracking measures. For this, we created rectangular AOI around the two numbers within the 32 word problems (height of 136 pixels, width of 40 pixels wide) in the eye-tracking software. The two numbers were always located in the second (y-coordinates of 246 and 382 pixels) and third line (y-coordinates of 386 and 522 pixels), but the location within these lines varied (x-coordinates differed depending on the word problem). On top of static and dynamic eye-tracking measures, we also recorded RT and ER.

We chose not to treat the reading of the question (i.e., first or last fixation on the question) as a the starting time point of measurement because reading might be non-linear (i.e., eye movements moving back and forth) and because of our hypothesis that word-problem solving does not necessarily follow a strictly sequential process. That is, by using an exclusion criterion for all stimuli where participants have not read the entire question, we might have risked considerable data loss. Furthermore, judgments about solvability and overall comprehension rely on integrating information from all parts of the problem.

Moreover, based on our study manipulation, we cannot determine when people recognize that a problem is non-solvable, nor can we separate how many eye movements happen before or after this time point. This would require a different type of experimental manipulation and follow-up research.

After extracting the seven dependent eye-tracking variables (static: FD, RD, FC, and RC; dynamic: NN, TT, and TN) and the two performance measures (RT and ER) from the EyeLink Data Viewer, data preprocessing was performed in the statistical computing software R (R Core Team, 2021). Missing and incorrect responses led to exclusion of the respective trial for the analysis of all dependent measures except ER. Moreover, all parallel word problems constructed from the same story were excluded if less than 33% of the solvable or non-solvable versions were correctly answered, because this indicates that the word problem leads to misunderstandings or is too difficult.

As for the data preprocessing, we used the statistical computing software R (R Core Team, 2021) for the data analysis, including the R packages dplyr (Wickham et al., 2023b), ggplot2 (Wickham et al., 2023a), ggpubr (Kassambara, 2023), and lmerTest (Kuznetsova et al., 2017). The anonymized datasets and all R scripts for the data preprocessing and analysis can be found on the Open Science Framework (https://​osf.​io/​nvdhq/​).

All 63 participants completed the entire study consisting of 32 word problems, so that all datasets were included in the data analysis. There were no missing responses. Incorrect responses led to exclusion of 469 out of 2016 trials (i.e., 23.26%) for the analysis of FD, FC, RD, RC, NN, TT, TN and RT (but not of ER). More precisely, 353 out of 1008 non-solvable trials (i.e., 35.02%) and 116 out of 1008 solvable trials (i.e., 11.51%) were incorrectly answered and therefore excluded. As shown in Supplementary Material H, we looked at accuracies for all word problems separately and checked whether any of the eight stories was frequently misunderstood and thus incorrectly responded to in more than 33% of the trials either within the two solvable or within the two non-solvable versions. This was the case for the “Cinema” story, where only 7.14% of the non-solvable versions were answered (see Table H1), so that all four “Cinema” word problems were excluded from all further analyses. For a better overview of all dependent variables (FD, FC, RD, RC, NN, TT, TN, RT, and ER) for each participant and word problem, we illustrated the means for each combination of solvability and operation (see Supplementary Material I). Note that difficulty (carry/borrow and non-carry/non-borrow) was counterbalanced between but not within these combinations.

The pattern of all static eye-tracking measures (FD, FC, RD, and RC) for the numbers (AOI) in correctly answered non-solvable word problems was similar. In all four variables, a significant interaction effect revealed that, as compared to the switch from addition to subtraction and from simple to complex, there was an overadditive effect when switching from simple addition to complex subtraction (see Tables 4, 5, 6, and 7). This crossover interaction is illustrated in Fig. 1. Moreover, the baseline of all static eye-tracking measures varied between participants, as reflected by significant random intercepts in all four LMMs. Because of the significant interaction effect, lower-level effects (i.e., main effects operation and difficulty) were not tested but remained in the LMM. As compared to the reference category of simple addition problems, descriptively less attention was drawn to simple subtraction and complex addition problems. Note that the results are similar for all four static eye-tracking variables, because FD, FC, RD, and RC are not independent from one another.

The pattern of all dynamic eye-tracking measures (NN, TT, and TN) in correctly answered non-solvable word problems was similar to the pattern of all static eye-tracking measures (FD, FC, RD, and RC). In all three variables, a significant interaction effect revealed that, as compared to the switch from addition to subtraction and from simple to complex, there was an overadditive effect when switching from simple addition to complex subtraction (see Tables 8, 9, and 10). This crossover interaction is illustrated in Fig. 2. Moreover, the baseline of all dynamic eye-tracking measures varied between participants, as reflected by significant random intercepts in all three LMMs. Because of the significant interaction effect, lower-level effects (i.e., main effects operation and difficulty) were not tested but remained in the LMM. As compared to the reference category of simple addition problems, descriptively less transitions were made in simple subtraction and complex addition problems. Note that, opposed to the four static eye-tracking variables, the results for the three dynamic eye-tracking variables are independent from one another.

A significant interaction effect between operation and difficulty was found for RT in correctly answered non-solvable word problems. This effect revealed that, as compared to the switch from addition to subtraction and from simple to complex, participants responded overadditively slower to complex subtraction than to simple addition problems (see Table 11). Moreover, the baseline of RT varied between participants, as reflected by significant random intercepts in the LMM. Because of the significant interaction effect, lower-level effects (i.e., main effects operation and difficulty) were not tested but remained in the LMM. Responses were descriptively slower in non-solvable simple subtraction than simple addition word problems. Further, responses were descriptively faster in non-solvable complex addition than simple addition word problems. RT are illustrated in Fig. 3A.

No significant interaction between operation and difficulty was detected for ER in non-solvable word problems. However, a significant main effect of operation revealed that significantly more errors were made in non-solvable subtraction than addition word problems (see Table 12). Moreover, the baseline of ER varied between participants, as reflected by significant random intercepts in the LMM. ER are illustrated in Fig. 3B.

In the present study, both behavioral and eye-tracking measures suggest simultaneous number and text processing. However, as revealed by an interaction of difficulty (carry/borrow vs. non-carry/non-borrow) and operation (addition vs. subtraction), the effectiveness of this common processing strategy depends on the type of problem, indicating that different strategies may be employed in various contexts. Please note that in all measures, interindividual variability influenced the results, as indicated by significant random participant intercepts.

The disordinal interaction between operation and difficulty in arithmetic word problems reflects that numbers start to be processed before the text is fully processed and before it is understood whether the problem is solvable or non-solvable. This suggests an integration of number and text processing, and it is possible that even more processes are involved simultaneously in word-problem solving. Our findings are supported by Zhou et al. (2018), who found evidence in neural correlates that, contrary to the view of a dissociation of text and number processing as shown by neuroimaging studies, the dissociation does not imply a complete separation of number processing from text processing due to the involvement of the semantic system. Nevertheless, these studies did not investigate the neural correlates of the carry/borrow effect within word problems. Our results provide compelling evidence for a magnitude-based mental representation, as supported by Bergqvist and Österholm (2010), whose model incorporates a cyclic component that enables it to update mental representations. Moreover, our findings are in line with conclusions drawn from pure text comprehension studies (without numerical information and mathematical operations to be performed), claiming that mental representations are built up in an incremental manner, allowing for regular updates during the reading process (Gernsbacher et al., 1998). This suggests that the mental representation of word problems is delicate, using an economical and effective parallel method for managing cognitive resources. This finding provides evidence against the sequential model proposed by Kintsch and Greeno (1985), who argue for a separation of text and number processing. According to their sequential model, a mental representation of a word problem is built by processing the surface and understanding the text semantics, and, in the next step, extending this text model by a problem model to select a fitting schema, fill it in with numbers, and finally carry out the calculation. According to this model, simply reading a non-solvable word problem should be sufficient to recognize that it is non-solvable, and a calculation should not even be initiated if the text of a non-solvable word problem is correctly understood. Notably, the significant interaction between difficulty and operation in non-solvable problems highlights that number manipulation influenced the solving process, which would not be the case if participants first read the text before initiating a calculation. Note that to rule out potential influences of the exact text and used story, all problems were standardized for word count, letter count, and word frequency (Roth, 2021). Thus, although the exact text and used story was counterbalanced within solvability and operation but not within difficulty (see Supplementary Materials A and I), it is highly unlikely that the observed interaction is due to textual differences.

Other models which try to explain how word problems are solved account for numbers in various ways but do not fully clarify what occurs in certain scenarios. The schema model and the situation model suggest that numbers are not necessarily processed from the beginning of the solving process. The SECO model examines how numbers function within word problems, attributing importance to the role and meaning of numbers as they semantically relate to the mental representation in word problems. The semantic relatedness was intentionally omitted from our study because the research aimed to explore number processing in a more abstract sense, specifically how arithmetic operations like carrying or borrowing influence problem-solving, without considering the contextual or semantic significance of the numbers involved. Our results suggest a more generalized role for numbers in problem-solving, indicating that numbers may play an early role in cognitive processing, even when they are not directly connected to semantic meanings.

To explain the disordinality of the observed interaction, we propose that the solving process is relatively adaptable and can shift based on the perceived or actual difficulty of the problems. Following the model proposed by Bergqvist and Österholm (2010), the initial reading (possibly skimming) of a non-solvable problem leads to a mental representation of the content. Based on this mental representation, a suitable strategy is chosen: re-reading, calculating, or ceasing the solving process. In some cases, individuals use the initial mental representation to label the problem as “solvable” and calculate or as “non-solvable” and cease the solving process, without re-reading the text. Such processes are supported by the fact that some behavioral carry or borrow effects are weaker than in previous studies. However, in other cases, individuals may opt to re-read the text, which might result in a modification of their initial mental representation (i.e., from a solvable to a non-solvable representation). In such cases, an extra step is necessary to correctly construct a mental representationand identify simple non-solvable problems as “non-solvable.” This additional step necessitates resources, which manifest as fixations, regressions, and transitions. Crucially, this process seems to be moderated by the problem’s perceived difficulty, aligning with the findings by Doz et al. (2023) and speaking against mutually exclusive strategies and instead for calculation already happening during reading and re-reading, most likely depending on the perceived difficulty of the problems.

The above reasoning explains well why there might be a borrow effect (as reflected by more fixations, regressions, and transitions) in non-solvable subtraction word problems. Namely, when a problem is non-solvable and involves borrowing, individuals might need more cognitive resources to confirm its non-solvability. Following the same reasoning, one would expect a similar pattern in addition problems, specifically a carry effect in non-solvable addition word problems. However, the observed crossover interaction reveals a descriptively reversed trend in addition, such that less attention is drawn to simple (non-carry) than to difficult (carry) non-solvable addition problems. One possible explanation is that individuals employ a direct translation strategy (e.g., Hegarty et al., 1995) when they perceive both the text and numbers as easy and initiate a calculation as it requires minimal cognitive resources (i.e., in the case of the non-carry addition problems, representing the easiest of the four problem types in the current study). They might then revise their problem representationor abandon it, depending on the complexity of the problem, which is defined as the combination of factors resulting from carry/non-carry in combination with addition/subtraction, along with other factors such as solvability and non-solvability. Nonetheless, even in case of abandonment of the calculation, a mismatch with the initial mental representationis recognized, indicating at least partial integration of number processing and text processing. The decision of whether to calculate may be influenced not only by motivational aspects but also by the simple act of “giving up” (Bergqvist & Österholm, 2010). Specifically, on a response (decision) level, participants may lower their threshold for answering that the presented word problem is non-solvable if the potential calculation is deemed as relatively complicated. In this case, it would be more likely and faster that participants abandon the calculation and respond that the problem is non-solvable. This would lead to less eye movements on the numbers in difficult than in easy non-solvable word problems. This notion is supported by the fact that the need for carrying is usually recognized relatively early during the encoding of the problem (Moeller et al., 2011b), so individuals could choose to capitulate relatively early.

While this explanation holds well for the eye-tracking data from addition word problems and the carry effect, it is, however, important to note that this explanation does not account for all findings of the current study. Namely, it does not account for the results of subtraction word problems and the borrow effect, where the more difficult arithmetic condition leads to more eye-movements and no such tendencies to abandon calculation or “giving up” can be observed. While both explanations (i.e., accounting for the addition findings and the carry effect on the one hand, and accounting for the subtraction findings and the borrow effect on the other hand) are possible and in line with some previous theoretical assumptions, we have no convincing explanation of why one explanation seems to be true for addition and the other for subtraction.

A different explanation for the reversed carry effect in static measures (i.e., fixations and regressions) for addition problems as reflected by the significant interactions might be addition is the default operation for many participants. That is, when faced with two numbers and before knowing what operation is actually required, addition might be more likely to be performed as compared to subtraction or other arithmetic operations. This assumption can be derived from, on the one hand, participants getting more involved in simple addition word problems than in complex ones (as reflected by more fixations and regressions in non-carry problems than in carry problems), but, on the other hand, no stronger involvement in simple subtraction word problems than in complex ones. Importantly, if an addition was performed with the two numbers in the subtraction problems, half of them would be carry problems and half of them would be non-carry problems. Thus, addition as a default operation would explain that there is on average no difference between borrow problems and non-borrow problems.

Importantly, it also remains unclear whether difficulty had an effect apart from its interaction with operation. As the interaction was significant and describes the eye movement patterns more precisely, the main effects of difficulty and operation were not even tested within the LMMs for any of the four measures. The descriptive pattern of results is consistent with the explanatory approach developed above. Another argument that supports this idea is that the average RT for solvable problems, where calculation is necessary to provide a correct solution, is much higher than for non-solvable problems, where mere reading comprehension is enough to answer correctly. Overall, it can be concluded that there is evidence for interactive processing of text and numbers and for different strategies in addition and subtraction word problems. Indeed, numbers play a significant role in word problem solving and are processed even when not required. These arguments are supported by Orrantina and Múñez(2013), who found suggesting that an automatic, analog magnitude-based mental representation is routinely activated during word problem solving. Our research not only supports these findings but provides even stronger evidence, as we even manipulated the calculation within the word problems.

Although the current study could clearly demonstrate that number processing takes part at an early stage in word problem solving, there are some limitations and future research questions to be considered. Firstly, the study employed a 2 (solvability) * 2 (operation) * 2 (difficulty) design, leading to eight word-problem types. Importantly, we decided to use each story for no more than four word problems. The reason for this was that if each story would have been repeated eight times, participants might have ceased to engage with the text thoroughly, potentially skipping most of the reading at some point due to familiarity, learning effects, and efficiency concerns. Thus, the design was not fully counterbalanced, but only across difficulty (carry/borrow), but only across solvability and operation. To mitigate the effects of this unbalanced design, we controlled for linguistic difficulty across the word problems. Therefore, we can argue that any differences due to linguistic difficulty arising from mean word frequencies, average sentence length, number of words, and unequal use of stories are absent or negligible. However, there might be other, not yet considered (or even unknown), factors that make word problems more difficult in certain conditions than in others.

Secondly, relational terms such as “more” or “less” should not be overlooked, as they are keywords for the arithmetic operations addition and subtraction, respectively. The evidence of interactive processing underscores the importance of closely examining integrative attention between text and numbers. Moreover, in several word problems, the second sentence already pointed towards an arithmetic operation (e.g., the words “in total” in the simple solvable “Museum” problem or in the complex solvable “Gardening” problem pointed towards subtraction), whereas in most other problems, this cue appeared only in the third sentence. Future studies should ensure the placement of keywords pointing towards a specific mathematical operation is consistent across word problems or is manipulated as a factor.

Thirdly, some used word problems in this study might have seemed ambiguous in terms of whether they were solvable or non-solvable. For instance, the simple non-solvable “Cinema” problem contained the information that two classes of 41 and 27 children and their teachers went to the cinema and asked about how many seats they needed. Since the number of teachers joining the trip to the cinema was not mentioned, the word problem was non-solvable, but some participants might have assumed that one teacher joined per class. This might explain why only 7.14% of the two (simple and complex) non-solvable versions of the “Cinema” word problems were identified as non-solvable. Moreover, in the non-solvable versions of the “Tennis,” “Museum,” “Busses,” and “Party” problems, some participants might have executed a calculation to find a minimum or maximum as an answer. For instance, if Sophie invites 24 and Mia invites 18 friends to their party, assuming that none of their friends declines the invitation, the maximum of 42 friends would come. The “Marbles,” “Gardening,” and “Fruits” problems, in contrast, might have been less ambiguous and thus more suitably constructed for the present investigation focusing on non-solvable word problems. In addition, cultural context might impact the perceived solvability of problems. Participants’ approaches to problem-solving and their assumptions about a problem’s solvability can be shaped by cultural norms and experiences, which influence their cognitive processing strategies (e.g., Rhodes et al., 2024). Therefore, we recommend that word problems should be specifically selected and tailored to fit cultural contexts.

Forthly, we should note that fixations and regressions of AOI, as well as transitions between parts of the text and numbers, do not capture the full calculation process. Rottmann and Schipper (2002) demonstrated that children tend to shift their gaze upwards or to an imaginary point when concentrating, which is potentially not even on the computer screen and therefore not recorded in eye-tracking. It is plausible that fixations, regressions, and transitions outside of AOI also indicate important cognitive processes in adults. Thus, a subsequent study could capture eye movements that do not target the word problem but instead points that are not located on the computer screen.

Lastly, we acknowledge that whether the question is fully read and whether attention is paid to the question early or late during the solving process may indeed play a crucial role in the word problem-solving process. However, to fully explore this aspect, we recommend conducting experiments specifically designed to investigate the unique role of the question within the processing of word problems. In general, our study does not allow to exactly locate when and how calculation and other processes take place and interact. We encourage other researchers to explore this question further by designing studies that can better isolate and identify processes and their interactions. Additionally, we hope that such future work will refine the existing theoretical framework.

The theoretical implication that calculation, or at least some form of number processing in word problems, occurs in parallel with text processing is profound. It challenges traditional models that suggest a sequential approach, where text processing (including reading comprehension) and number processing (including potential calculation) are distinct, subsequent phases. This finding supports a more integrated model, where number processing and text processing are not discrete, but interwoven from the outset. Ultimately, we require a new theoretical model that considers that number processing, including calculation steps, is a dynamic component of the word problem-solving process which varies according to the type of problem (i.e., solvability, operation, and arithmetic difficulty) and potentially its formulation as well.