Chaojun Xiao 肖朝军


Chief Scientist for Large Language Models
ModelBest (面壁智能)

Contact

  xcjthu [at] gmail [dot] com
  Github
  Google Scholar

About me

Hi! I am Chief Scientist for Large Language Models at ModelBest (面壁智能). I received my Ph.D. and bachelor's degree from Tsinghua University. At Tsinghua, I was advised by Prof. Zhiyuan Liu and Prof. Maosong Sun in the Natural Language Processing Lab (THUNLP).

My research centers on building efficient large language models: achieving stronger capabilities with fewer parameters, less computation, and better data. I work on four closely connected directions:

  • Capability density. I aim to understand and improve capability per parameter, making LLMs more capable and more efficient. [Densing Law]
  • Efficient architectures. I design efficient architectures, especially sparse attention, to make long-context intelligence practical under limited compute and memory. [InfLLM, InfLLM-V2, NOSA, MiniCPM-SALA]
  • Post-training algorithms and recursive self-improvement (RSI). I study scalable, data-efficient post-training and automated systems toward recursive self-improvement. [AutoSFT, Rethinking OPD I, Rethinking OPD II]
  • Open-source models and data. I aim to make capable, efficient models and high-quality training data broadly accessible through the MiniCPM and UltraData series. [MiniCPM4, MiniCPM5, MiniCPM-SALA, UltraData-Math, UltraData-SFT]

PUBLICATIONS

2026

1.   Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law. Preprint 2026. [pdf] Ziqing Qiao, Xiaoyue Xu, Sikui Zhang, Yinuo Xu, Bingxiang He, Chaojun Xiao†, Xu Han.
2.   Cognitively Decoupled Meta-Induction for Out-of-Distribution Generalization. EMNLP 2026. SAC Highlight Award [pdf] Yuqian Wang, Yuanzheng Wang, Chaojun Xiao†, Ke Xu, Bingxiang He, Jinghan Tan, Zijun Chen, Lu Chen, Yi-Xin Huo, Maosong Sun.
3.   AutoSFT: An Agentic System for Automated Data Research in Supervised Fine-Tuning. Preprint 2026. [pdf] Bingxiang He, Pengqi Sun, Shihao Peng, Shaoxuan Li, Huiyu Xiao, Hongjian Shi, Zixuan Fu, Ziqing Qiao, Yuxin Zuo, Cheng Qian, Xinqiang Yu, Jiayi Li, Xiusi Chen, Yizhen Song, Xiangyang Wang, Iofai Lei, Yudong Wang, Ning Ding, Zhiyuan Liu, Xiaozhi Wang, Chaojun Xiao†.
4.   Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution. Preprint 2026. [pdf] Pengcheng Huang, Zhenghao Liu, Shixin Ren, Shi Yu, Sijia Yao, Yukun Yan, Chaojun Xiao†, Maosong Sun.
5.   Rethinking On-Policy Distillation of Large Language Models II: One Training Example. Preprint 2026. [pdf] Zixuan Fu, Bingxiang He, Yuxin Zuo, Haohuan Huang, Jinqian Zhang, Ruhang Xiao, Cheng Qian, Qinyu Luo, Huan-ang Gao, Yudong Wang, Zhiyuan Liu, Ning Ding, Chaojun Xiao†.
6.   KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning. COLM 2026. [pdf] Cheng Gao, Cheng Huang, Kangyang Luo, Ziqing Qiao, Shuzheng Si, Huimin Chen, Chaojun Xiao†, Maosong Sun.
7.   Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. TMLR 2026. [pdf] Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao†, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, Ning Ding.
8.   Data Science and Technology Towards AGI Part I: Tiered Data Management. Preprint 2026. [pdf] Yudong Wang, Zixuan Fu, Hengyu Zhao, Chen Zhao, Chuyue Zhou, Xinle Lin, Hongya Lyu, Shuaikang Xue, Yi Yi, Yingjiao Wang, Zhi Zheng, Yuzhou Zhang, Jie Zhou, Chaojun Xiao†, Xu Han, Zhiyuan Liu, Maosong Sun.
9.   APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention. ACL 2026. [pdf] Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao†, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu.
10.   MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling. Preprint 2026. [pdf] MiniCPM Team (Chaojun Xiao† as technical lead).
11.   Rethinking the Role of Efficient Attention in Hybrid Architectures. EMNLP 2026. [pdf] Ziqing Qiao, Yinuo Xu, Chaojun Xiao†, Zhou Su, Zihan Zhou, Yingfa Chen, Xiaoyue Xu, Xu Han, Zhiyuan Liu.
12.   Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning. Preprint 2026. [pdf] Xiaoyue Xu, Sikui Zhang, Xiaorong Wang, Xu Han, Chaojun Xiao†.

2025

1.   Densing Law of LLMs. Nature Machine Intelligence 2025. [pdf] Chaojun Xiao, Jie Cai, Weilin Zhao, Biyuan Lin, Guoyang Zeng, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, Maosong Sun.
2.   H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs. Preprint 2025. [pdf] Cheng Gao, Huimin Chen, Chaojun Xiao†, Zhiyi Chen, Zhiyuan Liu, Maosong Sun.
3.   InfLLM-V2: Dense-sparse switchable attention for seamless short-to-long adaptation. Preprint 2025. [pdf] Weilin Zhao, Zihan Zhou, Zhou Su, Chaojun Xiao†, Yuxuan Li, Yanghao Li, Yudi Zhang, Weilun Zhao, Zhen Li, Yuxiang Huang, Ao Sun, Xu Han, Zhiyuan Liu.
4.   BlockFFN: Towards end-side acceleration-friendly mixture-of-experts with chunk-level activation sparsity. COLM 2025. [pdf] Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Yuxuan Li, Zhiyuan Liu, Maosong Sun.
5.   Document Segmentation Matters for Retrieval-Augmented Generation. ACL 2025 Findings. [pdf] Zhitong Wang, Cheng Gao, Chaojun Xiao†, Yufei Huang, Shuzheng Si, Kangyang Luo, Yuzhuo Bai, Wenhao Li, Tangjian Duan, Chuancheng Lv, Guoshan Lu, Gang Chen, Fanchao Qi, Maosong Sun.
6.   MiniCPM4: Ultra-efficient LLMs on end devices. Preprint 2025. [pdf] MiniCPM Team (Chaojun Xiao as technical lead).
7.   APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs. ACL 2025. [pdf] Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao†, Weilin Zhao, Sun Ao, Hao Zhou, Jie Zhou, Zhiyuan Liu, Maosong Sun.
8.   Ultra-FineWeb: Efficient data filtering and verification for high-quality LLM training data. Preprint 2025. [pdf] Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao†, Xu Han, Zhiyuan Liu.
9.   CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Sampling. Preprint 2025. [pdf] Zekai Qu, Yinxu Pan, Ao Sun, Chaojun Xiao†, Xu Han.
10.   NOSA: Native and Offloadable Sparse Attention. Preprint 2025. [pdf] Yuxiang Huang, Pengjie Wang, Jicheng Han, Weilin Zhao, Zhou Su, Ao Sun, Hongya Lyu, Hengyu Zhao, Yudong Wang, Chaojun Xiao, Xu Han, Zhiyuan Liu.

2024

1.   InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory. NeurIPS 2024. [pdf] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Maosong Sun.
2.   Fine-Grained Legal Argument-Pair Extraction via Coarse-Grained Pre-training. COLING 2024. [pdf] Chaojun Xiao, Yutao Sun, Yuan Yao, Xu Han, Wenbin Zhang, Zhiyuan Liu and Maosong Sun.
3.   Exploring the Benefit of Activation Sparsity in Pre-training. ICML 2024. [pdf] Zhengyan Zhang, Chaojun Xiao, Qiujieli Qin, Yankai Lin, Zhiyuan Zeng, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Jie Zhou.
4.   Configurable Foundation Models: Building LLMs from a Modular Perspective. Preprint 2024. [pdf] Chaojun Xiao, Zhengyan Zhang, Chenyang Song, Dazhi Jiang, Feng Yao, Xu Han, Xiaozhi Wang, Shuo Wang, Yufei Huang, Guanyu Lin, Yingfa Chen, Weilin Zhao, Yuge Tu, Zexuan Zhong, Ao Zhang, Chenglei Si, Khai Hao Moo, Chenyang Zhao, Huimin Chen, Yankai Lin, Zhiyuan Liu, Jingbo Shang, Maosong Sun.
5.   Enhancing Legal Case Retrieval via Scaling High-quality Synthetic Query-Candidate Pairs. EMNLP 2024. [pdf] Cheng Gao, Chaojun Xiao*, Zhenghao Liu, Huimin Chen, Zhiyuan Liu, Maosong Sun.

2023

1.   Plug-and-play document modules for pre-trained models. ACL 2023. [pdf] Chaojun Xiao, Zhengyan Zhang, Xu Han, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Xiangyang Li, Zhonghua Li, Zhao Cao, Maosong Sun.
2.   Plug-and-play knowledge injection for pre-trained language models. ACL 2023. [pdf] Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang, Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou.
3.   UPRec: User-Aware Pre-training for Recommender Systems. AI Open. [pdf] Chaojun Xiao, Ruobing Xie, Yuan Yao, Zhiyuan Liu, Maosong Sun, Xu Zhang, Leyu Lin.
4.   Variator: Accelerating pre-trained models with plug-and-play compression modules. EMNLP 2023 Findings. [pdf] Chaojun Xiao, Yuqi Luo, Wenbin Zhang, Pengle Zhang, Xu Han, Yankai Lin, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou.
5.   MUSER: A Multi-View Similar Case Retrieval Dataset. CIKM 2023 Resources. Best Resource Paper Honorable Mention [pdf] Qingquan Li, Yiran Hu, Feng Yao, Chaojun Xiao, Zhiyuan Liu, Maosong Sun, Weixing Shen.

2022

1.   LEVEN: A Large-Scale Chinese Legal Event Detection Dataset. ACL Findings 2022. Feng Yao, Chaojun Xiao*, Xiaozhi Wang, Zhiyuan Liu, Lei Hou, Cunchao Tu, Juanzi Li, Yun Liu, Weixing Shen, Maosong Sun.

2021

1.   Adversarial Language Games for Advanced Natural Language Intelligence. AAAI 2021. Long paper. Yuan Yao, Haoxi Zhong, Zhengyan Zhang, Xu Han, Xiaozhi Wang, Chaojun Xiao, Guoyang Zeng, Zhiyuan Liu, Maosong Sun.
2.   Equality before the Law: Legal Judgment Consistency Analysis for Fairness. Preprint. Yuzhong Wang, Chaojun Xiao, Shirong Ma, Haoxi Zhong, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, Maosong Sun.
3.   Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents. AI Open. SMP Best Paper Award. Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, Maosong Sun.
4.   CPM-2: Large-scale Cost-effective Pre-trained Language Models. AI Open. Zhengyan Zhang, Yuxian Gu, Xu Han, Shengqi Chen, Chaojun Xiao*, Zhenbo Sun, Yuan Yao, Fanchao Qi, Jian Guan, Pei Ke, Yanzheng Cai, Guoyang Zeng, Zhixing Tan, Zhiyuan Liu, Minlie Huang, Wentao Han, Yang Liu, Xiaoyan Zhu, Maosong Sun.

2020

1.   Denoising Relation Extraction from Document-level Distant Supervision. EMNLP 2020. Short paper. Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin and Leyu Lin.
2.   How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence. ACL 2020. Theme paper. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, Maosong Sun.
3.   More Data, More Relations, More context and More Openness: A Review and Outlook for Relation Extraction. AACL 2020. Long paper. Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou.
4.   JEC-QA: A Legal-Domain Question Answering Dataset. AAAI 2020. Long paper. (* indicates equal contribution). Haoxi Zhong, Chaojun Xiao*, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, Maosong Sun.
5.   Knowledge Transfer via Pre-training for Recommendation: A Review and Prospect. Frontiers in Big Data. (* indicates equal contribution). Zheni Zeng, Chaojun Xiao*, Yuan Yao, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun.

Before 2020

1.   Legal Judgment Prediction via Topological Learning. EMNLP 2018. Long Paper. Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, Maosong Sun.
2.   CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain. Preprint. (* indicates equal contribution). Chaojun Xiao*, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Tianyang Zhang, Xianpei Han, Zhen Hu, Heng Wang, Jianfeng Xu.
3.   CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction. Preprint. (* indicates equal contribution). Chaojun Xiao*, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, Jianfeng Xu.

PROJECTS

Open-Source Models

Datasets

EXPERIENCE

Ph.D. student

Department of Computer Science and Technology,
Tsinghua University, Beijing, China.
August 2020 - June 2025

Bachelor of Engineering

Department of Computer Science and Technology,
Tsinghua University, Beijing, China.
August 2016 - July 2020

High School

Pingchuan High School, Xingguo, Jiangxi, China.
August 2013 - July 2016

Reviewer

AAAI, WWW, COLING, ACL, EMNLP, NAACL, ACL ARR, NeurIPS.

TA

Towards Artificial General Intelligence, Tsinghua University.
2024
Object-Oriented Programming, Tsinghua University.
2020 - 2022
Media Programming, Tsinghua University.
2019 - 2021.

Awards

Promising Qingyuan Scholar, Chinese Association for Artificial Intelligence (CAAI)
2026
EMNLP 2026 SAC Highlight Award
2026
Fellowship of China National Postdoctoral Program for Innovative Talents
2025
Fellowship of Shui Mu Tsinghua Scholar Program
2025
Outstanding Doctoral Dissertation of Tsinghua University
2025

First Prize of Qian Weichang Award for Chinese Information Processing Science and Technology.
2024
First-class Tencent Rhino-Bird Elite Training Program Excellent Student.
2023
Second-class Overall Excellence Scholarship, Tsinghua University.
SMP Best Paper Award.
2022

Excellent Graduate, Beijing.
Excellent Graduate, Tsinghua University.
Excellent Graduate, Dept. of CS&T, Tsinghua University.
2020

First-class Price in Challenge Cup Contest, Beijing.
First-class Science and Technology Innovation Excellence Scholarship, Tsinghua University.
2019

First-class Price in Challenge Cup Contest, Tsinghua University.
First-class Overall Excellence Scholarship, Tsinghua University.
2018

First-class Overall Excellence Scholarship, Tsinghua University.
Gaotong Scholarship, Tsinghua University.
2017