Terminator: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning

Terminator is trained to predict the exact position of the first occurrence of the final answer of an LRM's CoT during online inference. An important challenge we overcome is to identify the exact position of the earliest final answer occurrence for a given CoT. Our training dataset is curated using the following pipeline for robust extraction of the answer position:

1. Collect CoTs from the LRM
2. For each CoT, we use a separate LRM to analyze the CoT for the final answer and extract its exact position
3. Given the token position index of the answer, we construct the labels: set all positions before the answer to 0 and all positions after to 1

Finally, Terminator is trained with binary cross-entropy loss over each position in the CoT. By using the above training strategy, Terminator learns to exit when the LRM has generated the final answer it will eventually commit to.

At inference time, Terminator makes a prediction for each generated token by the LRM. Terminator forces the end-of-thought token to terminate reasoning early by using the following rules:

• Predict 1 if the predicted confidence is above a fixed threshold (set to 0.7 by default)
• Count the number of predicted 1s in a sliding window of the most recent predictions (the window size is 10 by default)
• If the majority (>5 if the window size is 10) of predictions are 1, then the CoT is terminated

"Thinking tokens" such as wait, hmm, okay, and alternatively are associated with ongoing reasoning. We find that they exhibit measurable frequency shifts once the final answer is generated. The plots below show token rates before vs. after the answer for three such tokens. For example, hmm and okay occur more frequently before the answer, while another occurs more frequently after. Dot size reflects relative CoT length. Similar plots with different "thinking tokens" are shown in the appendix of our paper.

Figure 5: Token Usage Frequency Shift. "Thinking token" rates change depending on whether the final answer has been generated. Dot size reflects relative CoT length.

For easier problems like those in MATH-500, Terminator shows sharp transitions in predicted confidence at the exiting threshold, with good separation between the "still reasoning" and "answer generated" regimes. By manual inspection, we observe that there is a clearer transition to overthinking on these problems, which Terminator detects well.

Figure 6: Predicted Probabilities for MATH-500. Terminator's predicted probability stream for early-exiting on four randomly chosen samples from MATH-500.

Figure 7 zooms in on a single MATH-500 sample, showing the full CoT with Terminator's predicted probabilities overlaid. The predicted probability remains low throughout the reasoning phase and rises sharply once the final answer has been generated, triggering the early exit.

Figure 7: Predicted Probabilities for MATH-500. Terminator's predicted probabilities for early-exit on a randomly chosen sample from MATH-500. The beginning and the end are truncated for better visibility.

For harder problems like those in AIME 2025, the transition from productive reasoning to overthinking is less obvious, and Terminator's predicted probabilities do not show the same sharp transition seen on MATH-500. This is consistent with the observation that identifying a good exit position is more challenging on very hard tasks.

Figure 8: Predicted Probabilities for AIME 2025. Terminator's predicted probability stream for early-exiting on four randomly chosen samples from AIME 2025.

Figure 9 shows a case where Terminator struggles to find a clean exit position. The predicted probabilities oscillate near the threshold rather than making a decisive jump, reflecting the difficulty of the underlying problem. Despite this, Terminator still achieves strong performance on AIME 2025 overall.

Figure 9: Predicted Probabilities for AIME 2025. Terminator's predicted probabilities for early-exit on a randomly chosen sample from AIME 2025. The beginning and the end are truncated for better visibility.

For quick testing without spinning up a vLLM server, use the HuggingFace-native inference script included in each model repo:

python inference_hf.py --prompt "What is the sum of the first 100 natural numbers?"

Thinking content is streamed in dimmed text; the final answer is shown in bold. For best performance, the vLLM server is recommended.

The vLLM server exposes an OpenAI-compatible API, so you can use any OpenAI client library:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Terminator-Qwen3-8B",
    messages=[{"role": "user", "content": "What is 25 * 37?"}],
    temperature=0.6,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)

# Thinking content (chain-of-thought)
print(response.choices[0].message.reasoning)

# Final answer
print(response.choices[0].message.content)