We use a few cookies

Essential cookies keep the library working. With your permission, we'd also like to use analytics to see what's useful and improve the experience. Read our Privacy Policy or Cookie Policy.

Skip to main content
Home/e-Resources/Artificial Intelligence & Machine Learning/Strategies for Training Large Vocabulary Neural Language Models
Conference ProceedingsArtificial Intelligence & Machine Learning

Strategies for Training Large Vocabulary Neural Language Models

Checking your access to this record.

Abstract & Overview

Training neural network language models over large vocabularies is computationally costly compared to count-based models such as Kneser-Ney.We present a systematic comparison of neural strategies to represent and train large vocabularies, including softmax, hierarchical softmax, target sampling, noise contrastive estimation and self normalization.We extend self normalization to be a proper estimator of likelihood and introduce an efficient variant of softmax.We evaluate each method on three popular benchmarks, examining performance on rare words, the speed/accuracy trade-off and complementarity to Kneser-Ney.

Cite this record

Wenlin Chen, David Grangier, Michael Auli (2016). Strategies for Training Large Vocabulary Neural Language Models. Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/p16-1186

Copy in another style

Chen, W., Grangier, D., & Auli, M. (2016). Strategies for Training Large Vocabulary Neural Language Models. Annual Meeting of the Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/p16-1186
Author names are used as deposited, so check their order before you submit.

Metadata is harvested from open-access indexes and can be incomplete. Check a citation against the published version before relying on it — see our open access & attribution policy.