SeTox is a search-augmented framework for Chinese neologism toxicity detection. It combines a tool-use language model with live web search so the model can interpret emerging or implicit terms before making a safety decision.
- Dataset Hub: https://huggingface.co/datasets/thu-coai/SeTox
- Released Model Hubs: https://huggingface.co/thu-coai/SeTox-Qwen2.5-3B and https://huggingface.co/thu-coai/SeTox-Qwen2.5-7B
The repository contains code, examples, training configs, and the release data
package. The 3B and 7B model weights are available on Hugging Face.
Hugging Face dataset/model card drafts live under hf/, and upload notes are in
docs/huggingface_upload.md.
| Path | Purpose |
|---|---|
setox/ |
Search backends, prompt templates, inference runner, and metrics. |
scripts/run_inference.py |
Runs mock, offline, or search-augmented inference. |
scripts/evaluate.py |
Computes binary safety metrics from prediction JSON. |
scripts/validate_release.py |
Validates released data counts, schemas, and checksums. |
data/ |
Raw lexicon, SFT training data, evaluation sets, schema, and checksums. |
results/summary/ |
Released 3B/7B checkpoint metric summaries. |
train_code/ |
LLaMA-Factory SFT configs for the 3B and 7B checkpoints. |
hf/ |
Dataset and model card sources used for Hugging Face releases. |
pip install -r requirements.txtFor live search, copy .env.example to .env and set SERPER_API_KEY.
Supported search backends are:
serper: live web search through Serper.static: deterministic fixture search with--fixture_path.none: disables web search.
For offline smoke tests, use --mock or --search_backend none.
Run a lightweight pipeline check without loading a model:
python scripts/run_inference.py \
--mock \
--input_path examples/input.json \
--output_path outputs/example_predictions.json
python scripts/evaluate.py \
--input_path outputs/example_predictions.json \
--output_path outputs/example_metrics.jsonRun with a released model and live search:
python scripts/run_inference.py \
--model_path thu-coai/SeTox-Qwen2.5-3B \
--search_backend serper \
--input_path data/eval/neologism_test.json \
--output_path outputs/neologism_predictions.jsonUse thu-coai/SeTox-Qwen2.5-7B for the 7B checkpoint.
Evaluate a prediction file:
python scripts/evaluate.py \
--input_path outputs/neologism_predictions.json \
--output_path outputs/neologism_metrics.jsonTraining uses LLaMA-Factory full SFT. The released training data follows
ShareGPT/tool-call formatting and is indexed by data/train/dataset_info.json.
Install LLaMA-Factory in the training environment so llamafactory-cli is on
PATH, then run:
bash train_code/run.sh train_code/qwen2.5-7b.yaml
bash train_code/run.sh train_code/qwen2.5-3b.yamlThe default configs use public Qwen model ids and local output directories.
Adjust model_name_or_path, deepspeed, and output_dir for your environment.
The released checkpoint summaries are in
results/summary/qwen2.5-3b_text2_neologism.json and
results/summary/qwen2.5-7b_text2_neologism.json.
The cleaned reproducibility package includes:
data/raw/neologism_dct.json: 974 cleaned neologism records.data/train/train_tool.json: 995 tool-use training examples.data/train/train_normal.json: 600 direct training examples.data/eval/neologism_test.json: 624 neologism test examples.data/eval/general_test.json: 368 general safety test examples.
The released neologism file preserves the user-provided 974-record source file,
including seven duplicated source id values: 79, 132, 569, 591, 373,
427, and 730. We do not silently reindex the raw file; use list order when a
stable per-record index is needed. See data/README.md and data/schema.md for
details.
The tool-use training file is the exact final training snapshot and contains some
search-result noise; see docs/data_audit.md for the release-readiness audit.
The code and released data package in this repository are released under the MIT
License; see LICENSE.
The released model weights are fine-tuned from Qwen checkpoints and remain subject to the corresponding base model licenses:
thu-coai/SeTox-Qwen2.5-3B: based onQwen/Qwen2.5-3B-Instruct, under the Qwen Research License.thu-coai/SeTox-Qwen2.5-7B: based onQwen/Qwen2.5-7B-Instruct, under Apache-2.0.
Before publishing or modifying the release package, run:
python scripts/validate_release.py
python scripts/audit_training_noise.py
python -m compileall setox scriptsThe expected main checkpoint metric is within one percentage point of the paper result because live search and generation can introduce minor drift.
If you use SeTox, please cite the ACL paper:
- Paper: https://aclanthology.org/2026.acl-long.1602/
- DOI: https://doi.org/10.18653/v1/2026.acl-long.1602
@inproceedings{cui-etal-2026-setox,
title = "{S}e{T}ox: Search-enhanced Reasoning for {LLM}-based Toxicity Detection over {C}hinese Internet Buzzwords",
author = "Cui, Yiming and Zhang, Qinglin and Su, Xu and Min, Changyu and Hu, Shilin and Huang, Minlie",
editor = "Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2026",
address = "San Diego, California, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.acl-long.1602/",
doi = "10.18653/v1/2026.acl-long.1602",
pages = "33018--33034"
}