Tencent pretrain: A scalable and flexible toolkit for pre-Training models of different modalities

Recently, the success of pre-Training in text domain has been fully extended to vision, audio, and cross-modal scenarios. The proposed pre-Training models of different modalities are showing a rising trend of homogeneity in their model structures, which brings the opportunity to implement different pre-Training models within a uniform framework. In this paper, we present TencentPretrain, a toolkit supporting pre-Training models of different modalities. The core feature of TencentPretrain is the modular design. The toolkit uniformly divides pretraining models into 5 components: embedding, encoder, target embedding, decoder, and target. As almost all of common modules are provided in each component, users can choose the desired modules from different components to build a complete pre-Training model. The modular design enables users to efficiently reproduce existing pre-Training models or build brand-new one. We test the toolkit on text, vision, and audio benchmarks and show that it can match the performance of the original implementations.