Jiamin Li1, Yimin Jiang2, Yibo Zhu, Cong Wang1, Hong Xu2 1City University of Hong Kong, 2ByteDance Inc., 3The Chinese University of Hong Kong ATC 2023

Jiamin Li*, Hong Xu†, Yibo Zhu‡, Zherui Liu‡, Chuanxiong Guo‡, Cong Wang*, * City University of Hong Kong, † The Chinese University of Hong Kong, ‡ ByteDance Inc. Introduction

Adaptive Gating in Mixture-of-Experts based Language Models Jiamin Li 1, Qiang Su , Yitao Yang2, Yimin Jiang, Cong Wang1, Hong Xu2 1City University of Hong Kong, 2The Chinese University of Hong Kong [email protected], [email protected] [email protected], [email protected]

Jiamin Li*, Hong Xu†, Yibo Zhu‡, Zherui Liu‡, Chuanxiong Guo‡, Cong Wang*, * City University of Hong Kong, † The Chinese University of Hong Kong, ‡ ByteDance Inc. Introduction