8000 GitHub - wangyuenlp/underthinking · GitHub
[go: up one dir, main page]

Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 

Repository files navigation

TIP: Thought Switching Penalty for o1-like LLMs

This repository contains the open-source implementation of TIP (Thought Switching Penalty), a decoding strategy proposed in the paper:

"Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs"
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, Dong Yu
arXiv:2501.18585 [cs.CL]
Paper Link

🎯 Overview

TIP addresses the "underthinking" phenomenon in o1-like Large Language Models (LLMs), where models frequently switch between different reasoning thoughts without sufficiently exploring promising paths. This behavior leads to inadequate depth of reasoning and decreased performance, particularly on challenging mathematical problems.

🔧 How TIP Works

TIP implements a simple yet effective decoding strategy:

  1. Identify Thought Switching Tokens: Detect tokens that indicate transitions between reasoning approaches (e.g., "Alternatively", "alternatively")
  2. Apply Penalty: Reduce the probability of thought switching tokens when they appear too frequently within a specified window
  3. Encourage Depth: Promote deeper exploration of each reasoning path before considering alternatives

Algorithm Details

# Simplified TIP algorithm
def apply_tip_penalty(token_ids, logits, penalty=3.0, window=600):
    last_switch_pos = find_last_thought_switch(token_ids)
    tokens_since_switch = len(token_ids) - last_switch_pos
    
    if tokens_since_switch < window:
        for switch_token_id in thought_switch_tokens:
            logits[switch_token_id] -= penalty
    
    return logits

🚀 Quick Start

Installation

  1. Clone the repository:

    git clone <repository-url>
    cd underthinking
  2. Install dependencies:

    pip install -r requirements.txt
  3. Verify installation:

    python tip.py --help

Basic Usage

# Basic TIP inference with default settings
python tip.py \
    --input_file ./question.jsonl \
    --output_file ./results \
    --model_name_or_path Qwen/QwQ-32B-Preview \
    --enable_tip \
    --max_tokens 16000

# Advanced usage with custom TIP parameters
python tip.py \
    --input_file ./question.jsonl \
    --output_file ./results \
    --model_name_or_path Qwen/QwQ-32B-Preview \
    --enable_tip \
    --tip_penalty 5.0 \
    --tip_window 800 \
    --temperature 0.0 \
    --n_gpus 2

📋 Command Line Arguments

Core Arguments

Argument Type Default Description
--input_file str ./question.jsonl Path to input JSONL file
--output_file str ./output Base path for output file
--model_name_or_path str Qwen/QwQ-32B-Preview Model name or path

TIP-Specific Arguments

Argument Type Default Description
--enable_tip flag False Enable TIP penalty
--tip_penalty float 3.0 Penalty value for thought switching tokens
--tip_window int 600 Token window for applying penalty

Generation Parameters

Argument Type Default Description
--temperature float 0.0 Sampling temperature
--top_p float 1.0 Top-p sampling parameter
--max_tokens int 16000 Maximum tokens to generate
--n int 1 Number of samples per question

Hardware Configuration

Argument Type Default Description
--n_gpus int 1 Number of GPUs for tensor parallelism

📊 Input/Output Format

Input Format (JSONL)

{"question": "Solve the equation: 2x + 3 = 7"}
{"question": "What is the derivative of x^2 + 3x + 1?"}

Output Format (JSONL)

{
  "question": "Solve the equation: 2x + 3 = 7",
  "response": [
    {
      "content": "To solve 2x + 3 = 7...",
      "stop_reason": "stop"
    }
  ]
}

Key Findings

  1. Underthinking Correlation: Frequent thought switching correlates with incorrect responses
  2. Token Efficiency: Incorrect answers use 225% more tokens due to 418% more thought switches
  3. Performance Gains: TIP improves accuracy without requiring model fine-tuning

Model Requirements

  • Models should exhibit chain-of-thought reasoning
  • Support for custom logits processors
  • Compatible with vLLM inference engine

⚙️ Advanced Configuration

Custom Thought Switch Patterns

You can modify the thought switching detection by editing the identify_thought_switch_tokens() function:

# Add custom patterns
base_phrases = [
    'Alternatively,', 'alternatively,', 
    'On the other hand,', 'However,',
    'Let me try a different approach'
]

Performance Optimization

# Multi-GPU setup
python tip.py --n_gpus 4 --model_name_or_path <large-model>

# Memory optimization
python tip.py --max_model_len 16384 --gpu_memory_utilization 0.8

🔍 Troubleshooting

Common Issues

  1. CUDA Out of Memory:

    • Reduce --max_tokens or --max_model_len
    • Increase --n_gpus for tensor parallelism
    • Lower --gpu_memory_utilization
  2. Model Loading Errors:

    • Verify model path/name
    • Check HuggingFace authentication for gated models
    • Ensure sufficient disk space
  3. Performance Issues:

    • Use FP16 or quantization for faster inference
    • Adjust batch size based on available memory

Debug Mode

Enable detailed logging:

export PYTHONPATH=.
python -c "import logging; logging.basicConfig(level=logging.DEBUG)"
python tip.py --enable_tip --input_file small_test.jsonl

📚 Citation

If you use this implementation in your research, please cite:

@article{wang2025thoughts,
  title={Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs},
  author={Wang, Yue and Liu, Qiuzhi and Xu, Jiahao and Liang, Tian and Chen, Xingyu and He, Zhiwei and Song, Linfeng and Yu, Dian and Li, Juntao and Zhang, Zhuosheng and Wang, Rui and Tu, Zhaopeng and Mi, Haitao and Yu, Dong},
  journal={arXiv preprint arXiv:2501.18585},
  year={2025}
}

Note: This is an open-source reproduction of the TIP method for research purposes. The implementation maintains the core logic described in the paper while providing additional features for practical usage and reproducibility.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0