Reading + AI
LibrarySign in
Open the original paper on arXiv (new tab)

Training Large Language Models to Reason in a Continuous Latent Space

Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, Yuandong Tian

arXiv:2412.06769v4 · This workspace stays on this version. Check arXiv for newer versions

Some source details need review 1 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 117 math expressions, 0 rendering failures; 16 of 16 images available.

Compared 77 of 77 source passages against 18 of 18 PDF pages.

The PDF says “Using the example presented in Figure 4,” while the HTML says “Using the example presented in Figure [5](#S4.F5).” Check PDF page 6

16 of 16 figure, table and graphic images captured. Open original PDF

Abstract

Large language models (LLMs) are restricted to reason in the “language space”, where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue that language space may not always be optimal for reasoning. For example, most word tokens primarily ensure textual coherence and are not essential for reasoning, while some critical tokens require complex planning and pose huge challenges to LLMs. To explore the potential of LLM reasoning in an unrestricted latent space instead of using natural language, we introduce a new paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden state of the LLM as a representation of the reasoning state (termed “continuous thought”). Rather than decoding this into a word token, we feed it back to the LLM as the subsequent input embedding directly in the continuous space. This latent reasoning paradigm leads to the emergence of an advanced reasoning pattern: the continuous thought can encode multiple alternative next reasoning steps, allowing the model to perform a breadth-first search (BFS) to solve the problem, rather than prematurely committing to a single deterministic path like CoT. Coconut outperforms CoT on certain logical reasoning tasks that require substantial search during planning, and shows a better trade-off between accuracy and efficiency.

Large language models (LLMs) have demonstrated remarkable reasoning abilities, emerging from extensive pretraining on human languages (9; 1). While next token prediction is an effective training objective, it imposes a fundamental constraint on the LLM as a reasoning machine: the explicit reasoning process of LLMs must be generated in word tokens. For example, a prevalent approach, known as chain-of-thought (CoT) reasoning (44), involves prompting or training LLMs to generate solutions step-by-step using natural language. However, this is in stark contrast to certain human cognition results. Neuroimaging studies have consistently shown that the language network – a set of brain regions responsible for language comprehension and production – remains largely inactive during various reasoning tasks  (2; 31; 29; 30; 11). Further evidence indicates that human language is optimized for communication rather than reasoning (12).

A significant issue arises when LLMs use language for reasoning: the amount of reasoning required for each particular token varies greatly, yet current LLM architectures allocate nearly the same computing budget for predicting every token. Most tokens in a reasoning chain are generated solely for fluency, contributing little to the actual reasoning process. By contrast, some critical tokens require complex planning and pose huge challenges to LLMs. While previous work has attempted to fix these problems by prompting LLMs to generate succinct reasoning chains (27), or performing additional reasoning before generating some critical tokens (52), these solutions remain constrained within the language space and do not solve the fundamental problems. On the contrary, it would be ideal for LLMs to have the freedom to reason without any language constraints, and then translate their findings into language only when necessary.

Figure 1
Figure 1: A comparison of Chain of Continuous Thought (Coconut) with Chain-of-Thought (CoT). In CoT, the model generates the reasoning process as a word token sequence (e.g., [xi,xi+1,...,xi+j][x_{i},x_{i+1},...,x_{i+j}] in the figure). Coconut regards the last hidden state as a representation of the reasoning state (termed “continuous thought”), and directly uses it as the next input embedding. This allows the LLM to reason in an unrestricted latent space instead of a language space.

In this work we instead explore LLM reasoning in a latent space by introducing a novel paradigm, Coconut (Chain of Continuous Thought). It involves a simple modification to the traditional CoT process: instead of mapping between hidden states and language tokens using the language model head and embedding layer, Coconut directly feeds the last hidden state (a continuous thought) as the input embedding for the next token (Figure 1). This modification frees the reasoning from being within the language space, and the system can be optimized end-to-end by gradient descent, as continuous thoughts are fully differentiable. To enhance the training of latent reasoning, we employ a multi-stage training strategy inspired by 7, which effectively utilizes language reasoning chains to guide the training process.

Interestingly, our proposed paradigm leads to an efficient reasoning pattern. Unlike language-based reasoning, continuous thoughts in Coconut can encode multiple potential next steps simultaneously, allowing for a reasoning process akin to breadth-first search (BFS). While the model may not initially make the correct decision, it can maintain many possible options within the continuous thoughts and progressively eliminate incorrect paths through reasoning, guided by some implicit value functions. This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner, as seen in previous works (47; 21).

Experimentally, Coconut successfully enhances the reasoning capabilities of LLMs. For math reasoning (GSM8k, 6), using continuous thoughts is shown to be beneficial to reasoning accuracy, mirroring the effects of language reasoning chains. This indicates the potential to scale and solve increasingly challenging problems by chaining more continuous thoughts. On logical reasoning including ProntoQA (35), and our newly proposed ProsQA (Section 4) which requires stronger planning ability, Coconut and some of its variants even surpasses language-based CoT methods, while generating significantly fewer tokens during inference. We believe that these findings underscore the potential of latent reasoning and could provide valuable insights for future research.

Chain-of-thought (CoT) reasoning. We use the term chain-of-thought broadly to refer to methods that generate an intermediate reasoning process in language before outputting the final answer. This includes prompting LLMs (44; 23; 53), or training LLMs to generate reasoning chains, either with supervised finetuning (51; 49) or reinforcement learning (42; 22; 37; 48). 27 classified the tokens in CoT into symbols, patterns, and text, and proposed to guide the LLM to generate concise CoT based on analysis of their roles. Recent theoretical analyses have demonstrated the usefulness of CoT from the perspective of model expressivity (13; 28; 26). By employing CoT, the effective depth of the transformer increases because the generated outputs are looped back to the input (13). These analyses, combined with the established effectiveness of CoT, motivated our design of feeding the continuous thoughts back into the LLM as input embeddings. While CoT has proven effective for certain tasks, its autoregressive generation nature makes it challenging to mimic human reasoning on more complex problems (24; 21), which typically require planning and search. There are works that equip LLMs with explicit tree search algorithms (45; 47; 20), or train the LLM on search dynamics and trajectories (25; 14; 39). In our analysis, we find that after removing the constraint of a language space, a new reasoning pattern similar to BFS emerges, even though the model is not explicitly trained in this way.

Latent reasoning in LLMs. Previous works mostly define latent reasoning in LLMs as the hidden computation in transformers (46; 4). 46 constructed a dataset of two-hop reasoning problems and discovered that it is possible to recover the intermediate variable from the hidden representations. 4 further proposed to intervene the latent reasoning by “back-patching” the hidden representation. 36 discovered parallel latent reasoning paths in LLMs. Another line of work has discovered that, even if the model generates a CoT to reason, the model may actually utilize a different latent reasoning process. This phenomenon is known as the unfaithfulness of CoT reasoning (41; 40). To enhance the latent reasoning of LLMs, previous research proposed to augment it with additional tokens. 18 pretrained the model by randomly inserting a learnable <pause> tokens to the training corpus. This improves LLM’s performance on a variety of tasks, especially when followed by supervised finetuning with <pause> tokens. On the other hand, 32 further explored the usage of filler tokens, e.g., “...”, and concluded that they work well for highly parallelizable problems. However, 32 mentioned these methods do not extend the expressivity of the LLM like CoT; hence, they may not scale to more general and complex reasoning problems. 43 proposed to predict a planning token as a discrete latent variable before generating the next reasoning step. Recently, it has also been found that one can “internalize” the CoT reasoning into latent reasoning in the transformer with knowledge distillation (8) or a special training curriculum which gradually shortens CoT (7). 50 also proposed to distill a model that can reason latently from data generated with complex reasoning algorithms. These training methods can be combined to our framework, and specifically, we find that breaking down the learning of continuous thoughts into multiple stages, inspired by iCoT (7), is very beneficial for the training. Other work explores alternative architectures for latent reasoning, including looped transformers (16; 10), diffusion models in sentence embedding space (3). Recurrent memory transformers similarly pass information across segments through continuous memory embeddings updated at each recurrent step rather than through language tokens (5), with associative memory mechanisms later introduced to scale this recurrence to longer contexts (34). Different from these works, we focus on general multi-step reasoning tasks and aim to investigate the unique properties of latent reasoning in comparison to language space. In addition to reasoning tasks, 33 also explored using continuous space for multi-agent communication. Building on Coconut, 55 developed a theoretical framework demonstrating that continuous CoT can be more efficient than discrete CoT on certain tasks by encoding multiple reasoning paths in superposition states. Subsequently, 54 analyzed the training dynamics to explain how such superposition emerges under the Coconut training objective.

In this section, we introduce our new paradigm Coconut (Chain of Continuous Thought) for reasoning in an unconstrained latent space. We begin by introducing the background and notation we use for language models. For an input sequence x=(x1,...,xT)x=(x_{1},...,x_{T}), the standard large language model M\mathcal{M} can be described as:

Ht=Transformer(Et)H_{t}=\text{Transformer}(E_{t})
M(xt+1∣x≤t)=softmax(Wht)\mathcal{M}(x_{t+1}\mid x_{\leq t})=\text{softmax}(Wh_{t})

where Et=[e(x1),e(x2),...,e(xt)]E_{t}=[e(x_{1}),e(x_{2}),...,e(x_{t})] is the sequence of token embeddings up to position tt; Ht∈Rt×dH_{t}\in\mathbb{R}^{t\times d} is the matrix of the last hidden states for all tokens up to position tt; hth_{t} is the last hidden state of position tt, i.e., ht=Ht[t,:]h_{t}=H_{t}[t,:]; e(⋅)e(\cdot) is the token embedding function; WW is the parameter of the language model head.

Method Overview. In the proposed Coconut method, the LLM switches between the “language mode” and “latent mode” (Figure 1). In language mode, the model operates as a standard language model, autoregressively generating the next token. In latent mode, it directly utilizes the last hidden state as the next input embedding. This last hidden state represents the current reasoning state, termed as a “continuous thought”.

Special tokens <bot> and <eot> are employed to mark the beginning and end of the latent thought mode, respectively. As an example, we assume latent reasoning occurs between positions ii and jj, i.e., xi=x_{i}= <bot> and xj=x_{j}= <eot>. When the model is in the latent mode (i<t<ji<t<j), we use the last hidden state from the previous token to replace the input embedding, i.e., Et=[e(x1),e(x2),...,e(xi),hi,hi+1,...,ht−1]E_{t}=[e(x_{1}),e(x_{2}),...,e(x_{i}),h_{i},h_{i+1},...,h_{t-1}]. After the latent mode finishes (t≥jt\geq j), the input reverts to using the token embedding, i.e., Et=[e(x1),e(x2),...,e(xi),hi,hi+1,...,hj−1,e(xj),...,e(xt)]E_{t}=[e(x_{1}),e(x_{2}),...,e(x_{i}),h_{i},h_{i+1},...,h_{j-1},e(x_{j}),...,e(x_{t})]. It is worth noting that the last hidden states have been processed by the final normalization layer, so they are not too large in magnitude. M(xt+1∣x≤t)\mathcal{M}(x_{t+1}\mid x_{\leq t}) is not defined when i<t<ji<t<j, since the latent thought is not intended to be mapped back to language space. However, softmax(Wht)\mathrm{softmax}(Wh_{t}) can still be calculated for probing purposes (see Section 5).

Training Procedure. In this work, we focus on a problem-solving setting where the model receives a question as input and is expected to generate an answer through a reasoning process. We leverage language CoT data to supervise continuous thought by implementing a multi-stage training curriculum inspired by 7. As shown in Figure 2, in the initial stage, the model is trained on regular CoT instances. In the subsequent stages, at the kk-th stage, the first kk reasoning steps in the CoT are replaced with k×ck\times c continuous thoughts1^{1}, where cc is a hyperparameter controlling the number of latent thoughts replacing a single language reasoning step. Following 7, we also reset the optimizer state when training stages switch. We insert <bot> and <eot> tokens (which are not counted towards cc) to encapsulate the continuous thoughts.

Figure 2
Figure 2: Training procedure of Chain of Continuous Thought (Coconut). Given training data with language reasoning steps, at each training stage we integrate cc additional continuous thoughts (c=1c=1 in this example), and remove one language reasoning step. The cross-entropy loss is then used on the remaining tokens after continuous thoughts.

During the training process, we optimize the normal negative log-likelihood loss, but mask the loss on questions and latent thoughts. It is important to note that the objective does not encourage the continuous thought to compress the removed language thought, but rather to facilitate the prediction of future reasoning. Therefore, it’s possible for the LLM to learn more effective representations of reasoning steps compared to human language.

Training Details. Our proposed continuous thoughts are fully differentiable and allow for back-propagation. We perform n+1n+1 forward passes when nn latent thoughts are scheduled in the current training stage, computing a new latent thought with each pass and finally conducting an additional forward pass to obtain a loss on the remaining text sequence. While we can save any repetitive computing by using a KV cache, the sequential nature of the multiple forward passes poses challenges for parallelism. Further optimizing the training efficiency of Coconut remains an important direction for future research.

Inference Process. The inference process for Coconut is analogous to standard language model decoding, except that in latent mode, we directly feed the last hidden state as the next input embedding. A challenge lies in determining when to switch between latent and language modes. As we focus on the problem-solving setting, we insert a <bot> token immediately following the question tokens. For <eot>, we consider two potential strategies: a) train a binary classifier on latent thoughts to enable the model to autonomously decide when to terminate the latent reasoning, or b) always pad the latent thoughts to a constant length. We found that both approaches work comparably well. Therefore, we use the second option in our experiment for simplicity, unless specified otherwise.

In this section, we provide a proof of concept of the advantage of continuous latent space reasoning. On ProsQA, a new dataset that requires extensive planning ability, Coconut outperforms language space CoT reasoning. Interestingly, our analysis indicates that the continuous representation of reasoning can encode multiple alternative next reasoning steps. This allows the model to perform a breadth-first search (BFS) to solve the problem, instead of prematurely committing to a single deterministic path like language CoT.

We start by introducing the experimental setup (Section 4.1). By leveraging Coconut’s ability to switch between language and latent space reasoning, we are able to control the model to interpolate between fully latent reasoning and fully language reasoning and test their performance (Section 4.2). This also enables us to interpret the latent reasoning process as tree search (Section 4.3). Based on this perspective, we explain why latent reasoning can help LLMs make better decisions (Section 4.4).

Dataset. We introduce ProsQA (Proof with Search Question-Answering), a new logical reasoning dataset. A visualized example is shown in Figure 5. Each instance in ProsQA consists of a directed acyclic graph (DAG) of logical relationships between concepts, presented as natural language statements. The task requires models to determine logical relationships by finding valid paths through this graph, demanding sophisticated planning and search strategies. Unlike previous logical reasoning datasets like ProntoQA (35), ProsQA’s DAG structure introduces complex exploration paths, making it particularly challenging for models to identify the correct reasoning chain. More comprehensive details about the dataset construction and characteristics can be found in Appendix A.

Setup. We use a pre-trained GPT-2 model as the base model for all experiments. The learning rate is set to 1×10−41\times 10^{-4} while the effective batch size is 128. We train a Coconut model following the training procedure in Section 3. Since the maximum reasoning steps in ProsQA is 6, we set the number of training stages to N=6N=6 in the training procedure. In each stage, we train the model for 5 epochs, and stay in the last stage until the 50 epochs. The checkpoint with the best accuracy in the last stage is used for evaluation. As reference, we report the performance of (1) CoT: the model is trained with CoT data, and during inference, the model will generate a complete reasoning chain to solve the problem. (2) no-CoT: the model is trained with only the question and answer pairs, without any reasoning steps. During inference, the model will output the final answer directly.

To understand the properties of latent and language reasoning space, we manipulate the model to switch between fully latent reasoning and fully language reasoning, by manually setting the position of the <eot> token during inference. When we enforce Coconut to use kk continuous thoughts, the model is expected to output the remaining reasoning chain in language, starting from the k+1k+1 step. In our experiments, we test variants of Coconut on ProsQA with k∈{0,1,2,3,4,5,6}k\in\{0,1,2,3,4,5,6\}. Note that all these variants only differ in inference time while sharing the same model weights.

Metrics. We apply two sets of evaluation metrics. One of them is based on the correctness of the final answer, regardless of the reasoning process. It is also the main metric used in the later sections (Section 5.3). To enable fine-grained analysis on ProsQA, we define another metric on the reasoning process. We classify a reasoning chain into (1) Correct Path: The output is one of the shortest paths to the correct answer. (2) Longer Path: A valid path that correctly answers the question but is longer than the shortest path. (3) Hallucination: The path includes nonexistent edges or is disconnected. (4) Wrong Target: A valid path in the graph, but the destination node is not the one being asked. These four categories naturally apply to the output from Coconut (k=0k=0) and CoT, which generate the full path. For Coconut with k>0k>0 that outputs only partial paths in language (with the initial steps in continuous reasoning), we classify the reasoning as a Correct Path if a valid explanation can complete it. Also, we define Longer Path and Wrong Target for partial paths similarly. If no valid explanation completes the path, it’s classified as Hallucination. In no-CoT and Coconut with larger kk, the model may only output the final answer without any partial path, and it falls into (5) Correct Label or (6) Incorrect Label. These six categories cover all cases without overlap.

Figure 3
Figure 3: The accuracy of final answer (left) and reasoning process (right) of multiple variants of Coconut and baselines on ProsQA.

Figure 3 presents a comparative analysis of various reasoning methods evaluated on ProsQA. The model trained using CoT frequently hallucinates non-existent edges or outputs paths leading to incorrect targets, resulting in lower answer accuracy. In contrast, Coconut, which leverages continuous space reasoning, demonstrates improved accuracy as it utilizes an increasing number of continuous thoughts. Additionally, the rate of correct reasoning processes (indicated by “Correct Label” and “Correct Path”) significantly increases. At the same time, there is a notable reduction in instances of “Hallucination” and “Wrong Target,” issues that typically emerge when the model makes mistakes early in the reasoning process.

An intuitive demonstration of the limitations of reasoning in language space is provided by the case study depicted in Figure 5. As shown, models operating in language space often fail to plan ahead or backtrack. Once they commit to an incorrect path, they either hallucinate unsupported edges or terminate with irrelevant conclusions. In contrast, latent reasoning avoids such premature commitments by enabling the model to iteratively refine its decisions across multiple reasoning steps. This flexibility allows the model to progressively eliminate incorrect options and converge on the correct answer, ultimately resulting in higher accuracy.

To better understand Coconut, we probe the latent reasoning process by forcing the model to explicitly generate language reasoning steps following intermediate continuous thoughts (Figure 5). Using the example presented in Figure 5, at the initial reasoning step, the model must select which immediate child node of “Alex” to consider next, specifically from the set {“lempus”, “sterpus”, “zhorpus”, “grimpus”}. The distribution over these candidate next steps is visualized in Figure 5, left. In the subsequent reasoning step, these nodes expand further into an extended set of potential paths, including all grandchildren of “Alex” (Figure 5, right).

Figure 4
Figure 4: A case study of ProsQA. The model trained with CoT hallucinates an edge (Every yumpus is a rempus) after getting stuck in a dead end. Coconut (k=1) outputs a path that ends with an irrelevant node. Coconut (k=2) solves the problem correctly.

We define the predicted probability of a concept following continuous thoughts as a value function (Figure 5), estimating each node’s potential for reaching the correct target. Interestingly, the reasoning strategy employed by Coconut is not greedy search: while “lempus” initially has the highest value (0.33) at the first reasoning step (Figure 5, left), the model subsequently assigns the highest value (0.87) to “rorpus,” a child of “grimpus,” rather than following “lempus” (Figure 5, right). This characteristic resembles a breadth-first search (BFS) approach, contrasting sharply with the greedy decoding typical of traditional CoT methods. The inherent capability of continuous representations to encode multiple candidate paths enables the model to avoid making immediate deterministic decisions. Importantly, this tree search pattern is not limited to the illustrated example, but constitutes a fundamental mechanism underlying the consistent improvement observed with larger values of kk in Coconut.

Figure 6 presents an analysis of the parallelism in the model’s latent reasoning across the first and second thoughts. For the first thoughts (left panel), the cumulative values of the top-1, top-2, and top-3 candidate nodes are computed and plotted against their respective percentiles across the test set. The noticeable gaps between the three lines indicate that the model maintains significant diversity in its reasoning paths at this stage, suggesting a broad exploration of alternative possibilities. In contrast, the second thoughts (right panel) show a narrowing of these gaps. This trend suggests that the model transitions from parallel exploration to more focused reasoning in the second latent reasoning step, likely as it gains more certainty about the most promising paths.

Figure 6
Figure 6: Analysis of parallelism in the first two steps of the latent tree search. The three curves in each panel depict the cumulative value of the top-1, top-2, and top-3 candidate nodes.

Building upon the tree search perspective, we further examine why latent reasoning benefits planning tasks—specifically, why maintaining multiple candidate paths and postponing deterministic decisions enhances reasoning performance. Our hypothesis is that nodes explored in the early reasoning stages are inherently more challenging to evaluate accurately because they are farther from the final target nodes. In contrast, nodes positioned closer to potential targets, having fewer subsequent exploration possibilities, can be assessed accurately with higher confidence.

Figure 7
Figure 7: The correlation between the predicted value of correct/incorrect nodes and their heights.

To systematically test this, we define the height of a node as its shortest distance to any leaf node and analyze the relationship between node height and the model’s estimated value. Ideally, a correct node—one that can lead to the target node—should receive a high estimated value, whereas an incorrect node—one that cannot lead to the target node—should receive a low value. Empirical results across the test set (Figure 7) support our hypothesis: nodes with lower heights consistently receive more accurate and definitive probability evaluations. Conversely, nodes with greater heights exhibit more ambiguous evaluations, reflecting increased uncertainty.

These findings underscore the advantage of latent space reasoning. By delaying deterministic decisions and allowing exploration to proceed toward terminal states, latent reasoning significantly enhances the model’s ability to differentiate correct paths from incorrect ones, thereby improving performance on complex, planning-intensive tasks compared to traditional greedy methods.

After analyzing the promising parallel search pattern of Coconut, we validate the feasibility of LLM reasoning in a continuous latent space through more comprehensive experiments, highlighting its better reasoning efficiency over language space, as well as its potential to enhance the model’s expressivity with test-time scaling.

Table 1
MethodGSM8kProntoQAProsQA
Acc. (%)# TokensAcc. (%)# TokensAcc. (%)# Tokens
CoT42.9 ±\ \pm0.225.098.8 ±\ \pm0.892.577.5 ±\ \pm1.949.4
No-CoT16.5 ±\ \pm0.52.293.8 ±\ \pm0.73.076.7 ±\ \pm1.08.2
iCoT30.0∗2.299.8 ±\ \pm0.33.098.2 ±\ \pm0.38.2
Pause Token16.4 ±\ \pm1.82.277.7 ±\ \pm21.03.075.9 ±\ \pm0.78.2
Coconut (Ours)34.1 ±\ \pm1.58.299.8 ±\ \pm0.29.097.0 ±\ \pm0.314.2
- w/o curriculum14.4 ±\ \pm0.88.252.4 ±\ \pm0.49.076.1 ±\ \pm0.214.2
- w/o thought21.6 ±\ \pm0.52.399.9 ±\ \pm0.13.095.5 ±\ \pm1.18.2
- pause as thought24.1 ±\ \pm0.72.2100.0 ±\ \pm0.13.096.6 ±\ \pm0.88.2
Table 1: Results on three datasets: GSM8k, ProntoQA and ProsQA. Higher accuracy indicates stronger reasoning ability, while generating fewer tokens indicates better efficiency. ∗The result is from 7.
View original tableOriginal Table 1

Math Reasoning. We use GSM8k (6) as the dataset for math reasoning. It consists of grade school-level math problems. To train the model, we use a synthetic dataset generated by 8. We use two continuous thoughts for each reasoning step (i.e., c=2c=2). The model goes through 3 stages besides the initial stage. We then include an additional stage where still 3×c3\times c continuous thoughts are used as in the previous stage, but with all the remaining language reasoning chain removed. This handles the long-tail distribution of reasoning chains longer than 3 steps. We train the model for 6 epochs in the initial stage, and 3 epochs in each remaining stage.

Logical Reasoning. Logical reasoning involves the proper application of known conditions to prove or disprove a conclusion using logical rules. We use the ProntoQA (35) dataset, and our newly proposed ProsQA dataset, which is more challenging due to more distracting branches. We use one continuous thought for every reasoning step (i.e., c=1c=1). The model goes through 6 training stages in addition to the initial stage, because the maximum number of reasoning steps is 6 in these two datasets. The model then fully reasons with continuous thoughts to solve the problems in the last stage. We train the model for 5 epochs per stage.

For all datasets, after the standard schedule, the model stays in the final training stage, until reaching 50 epochs. We select the checkpoint based on the accuracy on the validation set. For inference, we manually set the number of continuous thoughts to be consistent with their final training stage. We use greedy decoding for all experiments.

We consider the following baselines: (1) CoT, and (2) No-CoT, which were introduced in Section 4. (3) iCoT (7): The model is trained with language reasoning chains and follows a carefully designed schedule that “internalizes” CoT. As the training goes on, tokens at the beginning of the reasoning chain are gradually removed until only the answer remains. During inference, the model directly predicts the answer. (4) Pause token (18): The model is trained using only the question and answer without a reasoning chain. However, different from No-CoT, special <pause> tokens are inserted between the question and answer, which provides the model with additional computational capacity to derive the answer. The number of <pause> tokens is set the same as continuous thoughts in Coconut.

We also evaluate some variants of Coconut: (1) w/o curriculum, which directly trains the model in the last stage. The model uses continuous thoughts to solve the whole problem. (2) w/o thought: We keep the multi-stage training, but don’t add any continuous latent thoughts. While this is similar to iCoT in the high-level idea, the exact training schedule is set to be consistent with Coconut, instead of iCoT, for a strict comparison. (3) Pause as thought: We use special <pause> tokens to replace the continuous thoughts, and apply the same multi-stage training curriculum as Coconut.

We show the overall results in Table 1. Using continuous thoughts effectively enhances LLM reasoning over the No-CoT baseline. For example, by using 6 continuous thoughts, Coconut achieves 34.1% accuracy on GSM8k, which significantly outperforms No-CoT (16.5%). We highlight several key findings below.

Figure 8
Figure 8: Efficiency comparison of reasoning space and Coconut with different cc.

“Chaining” continuous thoughts enhances reasoning. Language CoT proves to increase the effective depth of LLMs and enhance their expressiveness (13). Thus, generating more tokens serves as a way to inference-time scaling for reasoning (19; 38). This desirable property holds naturally for Coconut too. On GSM8k, Coconut outperformed other architectures trained with similar strategies, including Coconut (pause as thought) and Coconut (w/o thought). Particularly, it surpasses the latest baseline iCoT (7), which requires a more carefully designed training schedule.

Additionally, we experimented with adjusting the hyperparameter cc, which controls the number of latent thoughts corresponding to one language reasoning step (Figure 8, II). As we increased cc from 0 to 1 to 2, the model’s performance steadily improved.2^{2} This further validates the potential of continuous thoughts to scale up to harder problems. In two other synthetic tasks, we found that the variants of Coconut (w/o thoughts or pause as thought), and the iCoT baseline also achieve impressive accuracy. This indicates that the model’s computational capacity may not be the bottleneck in these tasks. In contrast, GSM8k involves more complex contextual understanding and modeling, placing higher demands on computational capability.

Figure 9
Figure 9: Decoding a continuous thought into language tokens in a math word problem. The decoded tokens correspond to intermediate variables that help solve the problem.

Continuous thoughts are efficient representations of reasoning. Compared to traditional CoT, Coconut generates fewer tokens while achieving higher accuracy on ProntoQA and ProsQA (Table 1). Although Coconut does not surpass CoT on GSM8k, it offers a superior trade-off between reasoning efficiency and accuracy (Figure 8, I). To illustrate this, we train a series of CoT models that progressively “internalize” (7) the initial m={0,1,2,3,ALL}m=\{0,1,2,3,\text{ALL}\} reasoning steps, and plot their accuracy versus the number of generated tokens (labeled as “language” in the figure). These models quickly lose accuracy as more reasoning steps are skipped. In contrast, by applying Coconut training strategy—replacing each language reasoning step with two continuous thoughts—the accuracy drop is substantially mitigated, maintaining higher performance even when fewer tokens are generated. Another interesting observation is that, when we decode the first continuous thought, it often corresponds to possible intermediate variables in the calculation (Figure 9). This also suggests that the continuous thoughts are more efficient representations of reasoning.

The LLM still needs guidance to learn latent reasoning. In the ideal case, the model should learn the most effective continuous thoughts automatically through gradient descent on questions and answers (i.e., Coconut w/o curriculum). However, from the experimental results, we found the models trained this way do not perform any better than no-CoT.

On the contrary, with the multi-stage curriculum, Coconut is able to achieve top performance across various tasks. The multi-stage training also integrates well with pause tokens (Coconut- pause as thought). Despite using the same architecture and similar multi-stage training objectives, we observed a small gap between the performance of iCoT and Coconut (w/o thoughts). The finer-grained removal schedule (token by token) and a few other tricks in iCoT may ease the training process. We leave combining iCoT and Coconut as future work. While the multi-stage training used for Coconut has proven effective, further research is definitely needed to develop better and more general strategies for learning reasoning in latent space, especially without the supervision from language reasoning chains.

In this paper, we introduce Coconut, a new paradigm for reasoning in continuous latent space. Experiments demonstrate that Coconut effectively enhances LLM performance across a variety of reasoning tasks. Reasoning in latent space gives rise to advanced emergent behaviors, where continuous thoughts can represent multiple alternative next steps. This enables the model to perform BFS over possible reasoning paths, rather than prematurely committing to a single deterministic trajectory as in language space CoT reasoning. Further research is needed to refine and scale latent reasoning to pretraining, which could improve generalization across a broader range of reasoning challenges. We hope our findings will spark continued exploration into latent reasoning, ultimately advancing the development of more capable machine reasoning systems.

Acknowledgement

The authors express their sincere gratitude to Jihoon Tack for his valuable discussions throughout the course of this work.

References

  1. Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1.
  2. Amalric and Dehaene (2019) M. Amalric and S. Dehaene A distinct cortical network for mathematical knowledge in the human brain. NeuroImage 189, pp. 19–31. Cited by: §1.
  3. Barrault et al. (2024) L. Barrault, P. Duquenne, M. Elbayad, A. Kozhevnikov, B. Alastruey, P. Andrews, M. Coria, G. Couairon, M. R. Costa-jussà, D. Dale, et al. Large concept models: language modeling in a sentence representation space. arXiv preprint arXiv:2412.08821. Cited by: §C.2, §2.
  4. Biran et al. (2024) E. Biran, D. Gottesman, S. Yang, M. Geva, and A. Globerson Hopping too late: exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406.12775. Cited by: §2.
  5. Bulatov et al. (2022) A. Bulatov, Y. Kuratov, and M. Burtsev Recurrent memory transformer. Advances in Neural Information Processing Systems 35. Cited by: §2.
  6. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §1, §5.1.
  7. Deng et al. (2024) Y. Deng, Y. Choi, and S. Shieber From explicit cot to implicit cot: learning to internalize cot step by step. arXiv preprint arXiv:2405.14838. Cited by: §C.1, §1, §2, §3, §5.2, §5.3, §5.3, Table 1.
  8. Deng et al. (2023) Y. Deng, K. Prasad, R. Fernandez, P. Smolensky, V. Chaudhary, and S. Shieber Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460. Cited by: §2, §5.1.
  9. Dubey et al. (2024) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §C.2, §1.
  10. Fan et al. (2024) Y. Fan, Y. Du, K. Ramchandran, and K. Lee Looped transformers for length generalization. arXiv preprint arXiv:2409.15647. Cited by: §2.
  11. Fedorenko et al. (2011) E. Fedorenko, M. K. Behr, and N. Kanwisher Functional specificity for high-level linguistic processing in the human brain. Proceedings of the National Academy of Sciences 108 (39), pp. 16428–16433. Cited by: §1.
  12. Fedorenko et al. (2024) E. Fedorenko, S. T. Piantadosi, and E. A. Gibson Language is primarily a tool for communication rather than thought. Nature 630 (8017), pp. 575–586. Cited by: §1.
  13. Feng et al. (2023) G. Feng, B. Zhang, Y. Gu, H. Ye, D. He, and L. Wang Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems 36. Cited by: §2, §5.3.
  14. Gandhi et al. (2024) K. Gandhi, D. Lee, G. Grand, M. Liu, W. Cheng, A. Sharma, and N. D. Goodman Stream of search (sos): learning to search in language. arXiv preprint arXiv:2404.03683. Cited by: §2.
  15. Geiping et al. (2025) J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein Scaling up test-time compute with latent reasoning: a recurrent depth approach. arXiv preprint arXiv:2502.05171. Cited by: §C.2.
  16. Giannou et al. (2023) A. Giannou, S. Rajput, J. Sohn, K. Lee, J. D. Lee, and D. Papailiopoulos Looped transformers as programmable computers. In International Conference on Machine Learning, pp. 11398–11442. Cited by: §2.
  17. Gladstone et al. (2025) A. Gladstone, G. Nanduru, M. M. Islam, P. Han, H. Ha, A. Chadha, Y. Du, H. Ji, J. Li, and T. Iqbal Energy-based transformers are scalable learners and thinkers. arXiv preprint arXiv:2507.02092. Cited by: §C.2.
  18. Goyal et al. (2023) S. Goyal, Z. Ji, A. S. Rawat, A. K. Menon, S. Kumar, and V. Nagarajan Think before you speak: training language models with pause tokens. arXiv preprint arXiv:2310.02226. Cited by: §2, §5.2.
  19. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §5.3.
  20. Hao et al. (2024) S. Hao, Y. Gu, H. Luo, T. Liu, X. Shao, X. Wang, S. Xie, H. Ma, A. Samavedhi, Q. Gao, et al. LLM reasoners: new evaluation, library, and analysis of step-by-step reasoning with large language models. arXiv preprint arXiv:2404.05221. Cited by: §2.
  21. Hao et al. (2023) S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992. Cited by: §1, §2.
  22. Havrilla et al. (2024) A. Havrilla, Y. Du, S. C. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravinskyi, E. Hambro, S. Sukhbaatar, and R. Raileanu Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642. Cited by: §2.
  23. Khot et al. (2022) T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal Decomposed prompting: a modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406. Cited by: §2.
  24. LeCun (2022) Y. LeCun A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review 62 (1), pp. 1–62. Cited by: §2.
  25. Lehnert et al. (2024) L. Lehnert, S. Sukhbaatar, P. Mcvay, M. Rabbat, and Y. Tian Beyond a*: better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402.14083. Cited by: §2.
  26. Li et al. (2024) Z. Li, H. Liu, D. Zhou, and T. Ma Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint arXiv:2402.12875. Cited by: §2.
  27. Madaan and Yazdanbakhsh (2022) A. Madaan and A. Yazdanbakhsh Text and patterns: for effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686. Cited by: §1, §2.
  28. Merrill and Sabharwal (2023) W. Merrill and A. Sabharwal The expresssive power of transformers with chain of thought. arXiv preprint arXiv:2310.07923. Cited by: §2.
  29. Monti et al. (2007) M. M. Monti, D. N. Osherson, M. J. Martinez, and L. M. Parsons Functional neuroanatomy of deductive inference: a language-independent distributed network. Neuroimage 37 (3), pp. 1005–1016. Cited by: §1.
  30. Monti et al. (2009) M. M. Monti, L. M. Parsons, and D. N. Osherson The boundaries of language and thought in deductive inference. Proceedings of the National Academy of Sciences 106 (30), pp. 12554–12559. Cited by: §1.
  31. Monti et al. (2012) M. M. Monti, L. M. Parsons, and D. N. Osherson Thought beyond language: neural dissociation of algebra and natural language. Psychological science 23 (8), pp. 914–922. Cited by: §1.
  32. Pfau et al. (2024) J. Pfau, W. Merrill, and S. R. Bowman Let’s think dot by dot: hidden computation in transformer language models. arXiv preprint arXiv:2404.15758. Cited by: §2.
  33. Pham et al. (2023) C. Pham, B. Liu, Y. Yang, Z. Chen, T. Liu, J. Yuan, B. A. Plummer, Z. Wang, and H. Yang Let models speak ciphers: multiagent debate through embeddings. arXiv preprint arXiv:2310.06272. Cited by: §2.
  34. Rodkin et al. (2024) I. Rodkin, Y. Kuratov, A. Bulatov, and M. Burtsev Associative recurrent memory transformer. arXiv preprint arXiv:2407.04841. Cited by: §2.
  35. Saparov and He (2022) A. Saparov and H. He Language models are greedy reasoners: a systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.01240. Cited by: §A.2, §1, §4.1, §5.1.
  36. Shalev et al. (2024) Y. Shalev, A. Feder, and A. Goldstein Distributional reasoning in llms: parallel reasoning processes in multi-hop reasoning. arXiv preprint arXiv:2406.13858. Cited by: §2.
  37. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §2.
  38. Snell et al. (2024) C. Snell, J. Lee, K. Xu, and A. Kumar Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314. Cited by: §5.3.
  39. Su et al. (2024) D. Su, S. Sukhbaatar, M. Rabbat, Y. Tian, and Q. Zheng Dualformer: controllable fast and slow thinking by learning with randomized reasoning traces. arXiv preprint arXiv:2410.09918. Cited by: §2.
  40. Turpin et al. (2024) M. Turpin, J. Michael, E. Perez, and S. Bowman Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems 36. Cited by: §2.
  41. Wang et al. (2022) B. Wang, S. Min, X. Deng, J. Shen, Y. Wu, L. Zettlemoyer, and H. Sun Towards understanding chain-of-thought prompting: an empirical study of what matters. arXiv preprint arXiv:2212.10001. Cited by: §2.
  42. Wang et al. (2024) P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui Math-shepherd: verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426–9439. Cited by: §2.
  43. Wang et al. (2023) X. Wang, L. Caccia, O. Ostapenko, X. Yuan, W. Y. Wang, and A. Sordoni Guiding language model reasoning with planning tokens. arXiv preprint arXiv:2310.05707. Cited by: §2.
  44. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, pp. 24824–24837. Cited by: §1, §2.
  45. Xie et al. (2023) Y. Xie, K. Kawaguchi, Y. Zhao, J. X. Zhao, M. Kan, J. He, and M. Xie Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems 36. Cited by: §2.
  46. Yang et al. (2024) S. Yang, E. Gribovskaya, N. Kassner, M. Geva, and S. Riedel Do large language models latently perform multi-hop reasoning?. arXiv preprint arXiv:2402.16837. Cited by: §2.
  47. Yao et al. (2023) S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan Tree of thoughts: deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36. Cited by: §1, §2.
  48. Yu et al. (2024a) F. Yu, L. Jiang, H. Kang, S. Hao, and L. Qin Flow of reasoning: efficient training of llm policy with divergent thinking. arXiv preprint arXiv:2406.05673. Cited by: §2.
  49. Yu et al. (2023) L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu Metamath: bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284. Cited by: §2.
  50. Yu et al. (2024b) P. Yu, J. Xu, J. Weston, and I. Kulikov Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023. Cited by: §2.
  51. Yue et al. (2023) X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen Mammoth: building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653. Cited by: §2.
  52. Zelikman et al. (2024) E. Zelikman, G. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. D. Goodman Quiet-star: language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629. Cited by: §1.
  53. Zhou et al. (2022) D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625. Cited by: §2.
  54. Zhu et al. (2025a) H. Zhu, S. Hao, Z. Hu, J. Jiao, S. Russell, and Y. Tian Emergence of superposition: unveiling the training dynamics of chain of continuous thought. arXiv preprint arXiv:2509.23365. Cited by: §2.
  55. Zhu et al. (2025b) H. Zhu, S. Hao, Z. Hu, J. Jiao, S. Russell, and Y. Tian Reasoning by superposition: a theoretical perspective on chain of continuous thought. arXiv preprint arXiv:2505.12514. Cited by: §2.

We provide some examples of the questions and CoT solutions for the datasets used in our experiments.

Graphic from the paper; its text follows
Text in this graphic

GSM8k

Question = "John cuts his grass to 2 inches. It grows .5 inches per month. When it gets to 4 inches he cuts it back down to 2 inches. It cost $100 to get his grass cut. How much does he pay per year?"
Steps = ["<<4-2=2>>", "<<2/.5=4>>", "<<12/4=3>>", "<<100*3=300>>"]
Answer = "300"

Graphic from the paper; its text follows
Text in this graphic

ProntoQA

Question = "Brimpuses are not luminous. Shumpuses are amenable. Each yumpus is a lorpus. Gorpuses are shumpuses. Each zumpus is a grimpus. Gorpuses are rompuses. Dumpuses are not floral. Lempuses are cold. Brimpuses are impuses. Every lorpus is floral. Every rompus is transparent. Grimpuses are muffled. Rompuses are yumpuses. Rompuses are wumpuses. Zumpuses are fast. Wumpuses are bitter. Every sterpus is orange. Each lorpus is a vumpus. Yumpuses are feisty. Each yumpus is a lempus. Gorpuses are snowy. Zumpuses are gorpuses. Every lorpus is a sterpus. Stella is a brimpus. Stella is a zumpus. True or false: Stella is not floral."
Steps = ["Stella is a zumpus. Zumpuses are gorpuses.", "Stella is a gorpus. Gorpuses are rompuses.", "Stella is a rompus. Rompuses are yumpuses.", "Stella is a yumpus. Each yumpus is a lorpus.", "Stella is a lorpus. Every lorpus is floral.", "Stella is floral."]
Answer = "False"

Graphic from the paper; its text follows
Text in this graphic

ProsQA

Question = "Every shumpus is a rempus. Every shumpus is a yimpus. Every terpus is a fompus. Every terpus is a gerpus. Every gerpus is a brimpus. Alex is a rempus. Every rorpus is a scrompus. Every rorpus is a yimpus. Every terpus is a brimpus. Every brimpus is a lempus. Tom is a terpus. Every shumpus is a timpus. Every yimpus is a boompus. Davis is a shumpus. Every gerpus is a lorpus. Davis is a fompus. Every shumpus is a boompus. Every shumpus is a rorpus. Every terpus is a lorpus. Every boompus is a timpus. Every fompus is a yerpus. Tom is a dumpus. Every rempus is a rorpus. Is Tom a lempus or scrompus?"
Steps = ["Tom is a terpus.", "Every terpus is a brimpus.", "Every brimpus is a lempus."]
Answer = "Tom is a lempus."

To construct the dataset, we first compile a set of typical entity names, such as “Alex” and “Jack,” along with fictional concept names like “lorpus” and “rorpus,” following the setting of ProntoQA (35). Each problem is structured as a binary question: “Is [Entity] a [Concept A] or [Concept B]?” Assuming [Concept A] is the correct answer, we build a directed acyclic graph (DAG) where each node represents an entity or a concept. The graph is constructed such that a path exists from [Entity] to [Concept A] but not to [Concept B].

Algorithm 1 describes the graph construction process. The DAG is incrementally built by adding nodes and randomly connecting them with edges. To preserve the validity of the binary choice, with some probability, we enforce that the new node cannot simultaneously serve as a descendant to both node 00 and 11. This separation maintains distinct families of nodes and balances their sizes to prevent model shortcuts.

After the graph is constructed, nodes without parents are assigned entity names, while other nodes receive concept names. To formulate a question of the form “Is [Entity] a [Concept A] or [Concept B]?”, we designate node 00 in the graph as [Entity], a leaf node labeled 11 as [Concept A], and a leaf node labeled 22 as [Concept B]. This setup ensures a path from [Entity] to [Concept A] without any connection to [Concept B], introducing a moderately complex reasoning path. Finally, to avoid positional biases, [Concept A] and [Concept B] are randomly permuted in each question.

Algorithm 1 Graph Construction for ProsQA
edges←{}edges\leftarrow\{\}
nodes←{0,1}nodes\leftarrow\{0,1\}
labels←{0:1,1:2}labels\leftarrow\{0:1,1:2\}
▹\triangleright Labels: 1 (descendant of node 0), 2 (descendant of node 1), 3 (both), 0 (neither).
groups←{0:{},1:{0},2:{1},3:{}}groups\leftarrow\{0:\{\},1:\{0\},2:\{1\},3:\{\}\}
idx←2idx\leftarrow 2
while idx<Nidx<N do ▹\triangleright For each new node, randomly add edges from existing nodes
n_in_nodes←poisson(1.5)n\_in\_nodes\leftarrow\text{poisson}(1.5)
rand←random()rand\leftarrow\text{random}()
if rand≤0.35rand\leq 0.35 then
candidates←groups[0]∪groups[1]candidates\leftarrow groups[0]\cup groups[1] ▹\triangleright Cannot be a descendant of node 1.
else if rand≤0.7rand\leq 0.7 then
candidates←groups[0]∪groups[2]candidates\leftarrow groups[0]\cup groups[2] ▹\triangleright Cannot be a descendant of node 0.
else
candidates←nodescandidates\leftarrow nodes
end if
n_in_nodes←min⁡(len(candidates),n_in_nodes)n\_in\_nodes\leftarrow\min(\text{len}(candidates),n\_in\_nodes)
weights←[depth_to_root(c)⋅1.5+1  ∀c∈candidates]weights\leftarrow[\text{depth\_to\_root}(c)\cdot 1.5+1\;\forall c\in candidates]
▹\triangleright Define sampling weights to prioritize deeper nodes.
▹\triangleright This way, the solution reasoning chain is expected to be longer.
in_nodes←random_choice(candidates,n_in_nodes,prob=weights/sum(weights))in\_nodes\leftarrow\text{random\_choice}(candidates,n\_in\_nodes,\text{prob}=weights/\text{sum}(weights))
cur_label←0cur\_label\leftarrow 0
for in_idx∈in_nodesin\_idx\in in\_nodes do
cur_label←cur_label∣labels[in_idx]cur\_label\leftarrow cur\_label\mid labels[in\_idx] ▹\triangleright Update label using bitwise OR.
edges.append((in_idx,idx))edges.\text{append}((in\_idx,idx))
end for
groups[cur_label].append(idx)groups[cur\_label].\text{append}(idx)
labels[idx]←cur_labellabels[idx]\leftarrow cur\_label
nodes←nodes∪{idx}nodes\leftarrow nodes\cup\{idx\}
idx←idx+1idx\leftarrow idx+1
end while
Table 2
# Nodes# EdgesLen. of Shortest Path# Shortest Paths
23.036.03.81.6
Table 2: Statistics of the graph structure in ProsQA.
View original tableOriginal Table 2

We show the size of all datasets in Table 3.

Table 3
DatasetTrainingValidationTest
GSM8k385,6205001319
ProntoQA9,000200800
ProsQA17,886300500
Table 3: Statistics of the datasets.
View original tableOriginal Table 3

We present a clock-time comparison to evaluate reasoning efficiency. The reported values represent the average inference time per test case (in seconds), with a batch size of 1, measured on an Nvidia A100 GPU. For the no-CoT and CoT baselines, we employ the standard generate method from the transformers3^{3} library. Our results show that clock time is generally proportional to the number of newly generated tokens, as detailed in Table 1.

Table 4
MethodGSM8kProntoQAProsQA
No-CoT0.030.030.08
CoT0.260.850.47
Coconut0.090.110.15
Table 4: Inference time (in seconds) comparison across tasks and methods.
View original tableOriginal Table 4

In Figure 8 (II), we present the performance of Coconut on GSM8k using c∈{0,1,2}c\in\{0,1,2\}. When experimenting with c=3c=3, we observe a slight performance drop accompanied by increased variance. Analysis of the training logs indicates that adding three continuous thoughts at once – particularly during the final stage transition – leads to a sharp spike in training loss, causing instability. Future work will explore finer-grained schedules, such as incrementally adding continuous thoughts one at a time while removing fewer language tokens, as in iCoT (7). Additionally, combining language and latent reasoning—e.g., generating the reasoning skeleton in language and completing the reasoning process in latent space—could provide a promising direction for improving performance and stability.

We experimented with Coconut on GSM8k using Llama 3.2-3B and Llama 3-8B (9) with c=1c=1. We train them for 3 epochs in Stage 0, followed by 1 epoch per subsequent stage. The results are shown in Table 5.

Table 5
Modelno-CoTCoconut (Ours)
Llama 3.2-3B26.031.7
Llama 3-8B42.243.6
Table 5: Experimental results of applying Coconut to larger Llama models. We report performance comparisons between models without CoT reasoning (no-CoT) and our proposed Coconut method.
View original tableOriginal Table 5

We observe consistent performance gains across both Llama 3.2-3B and Llama 3-8B models compared to the no-CoT baseline, though these improvements are not as pronounced as those previously demonstrated with GPT-2. One possible reason is that larger models have already undergone extensive language-focused pre-training, making the transition to latent reasoning more challenging.

We emphasize that the primary goal of this paper is to highlight the promising attributes of latent-space reasoning and to initiate exploration in this new direction. Universally surpassing language-based CoT likely requires significant research efforts dedicated to latent space pre-training. We are encouraged by recent progress in this area (15; 3; 17). While these recent models provide scalable methods for latent representation learning, the latent spaces have not yet been explicitly optimized for reasoning. Integrating these recent advancements with Coconut presents an exciting and promising avenue for future research.