We use a few cookies

Essential cookies keep the library working. With your permission, we'd also like to use analytics to see what's useful and improve the experience. Read our Privacy Policy or Cookie Policy.

Skip to main content
Home/e-Resources/Artificial Intelligence & Machine Learning/Distilling Knowledge Learned in BERT for Text Generation
Conference ProceedingsArtificial Intelligence & Machine Learning

Distilling Knowledge Learned in BERT for Text Generation

Checking your access to this record.

Abstract & Overview

Large-scale pre-trained language model such as BERT has achieved great success in language understanding tasks.However, it remains an open question how to utilize BERT for language generation.In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks.The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text generation performance.By leveraging BERT's idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation.Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization.Our proposed model also achieves new state of the art on IWSLT German-English and English-Vietnamese MT datasets.1

Cite this record

Yen-Chun Chen, Zhe Gan, Cheng Yu, Jingzhou Liu, Jun Liu (2020). Distilling Knowledge Learned in BERT for Text Generation. Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2020.acl-main.705

Copy in another style

Chen, Y., Gan, Z., Yu, C., Liu, J., & Liu, J. (2020). Distilling Knowledge Learned in BERT for Text Generation. Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2020.acl-main.705
Author names are used as deposited, so check their order before you submit.

Metadata is harvested from open-access indexes and can be incomplete. Check a citation against the published version before relying on it — see our open access & attribution policy.