Publication detail
i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models
BENEŠ, K. KESIRAJU, S. BURGET, L.
Original Title
i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models
Type
conference paper
Language
English
Original Abstract
We show an effective way of adding context information toshallow neural language models. We propose to use SubspaceMultinomial Model (SMM) for context modeling and we addthe extracted i-vectors in a computationally efficient way. Byadding this information, we shrink the gap between shallowfeed-forward network and an LSTM from 65 to 31 points of perplexityon the Wikitext-2 corpus (in the case of neural 5-grammodel). Furthermore, we show that SMM i-vectors are suitablefor domain adaptation and a very small amount of adaptationdata (e.g. endmost 5% of a Wikipedia article) brings asubstantial improvement. Our proposed changes are compatiblewith most optimization techniques used for shallow feedforwardLMs.
Keywords
language modeling, feed-forward models, subspacemultinomial model, domain adaptation
Authors
BENEŠ, K.; KESIRAJU, S.; BURGET, L.
Released
2. 9. 2018
Publisher
International Speech Communication Association
Location
Hyderabad
ISBN
1990-9772
Periodical
Proceedings of Interspeech
Year of study
2018
Number
9
State
French Republic
Pages from
3383
Pages to
3387
Pages count
5
URL
BibTex
@inproceedings{BUT155102,
author="Karel {Beneš} and Santosh {Kesiraju} and Lukáš {Burget}",
title="i-vectors in language modeling: An efficient way of domain adaptation for feed-forward models",
booktitle="Proceedings of Interspeech 2018",
year="2018",
journal="Proceedings of Interspeech",
volume="2018",
number="9",
pages="3383--3387",
publisher="International Speech Communication Association",
address="Hyderabad",
doi="10.21437/Interspeech.2018-1070",
issn="1990-9772",
url="https://www.isca-speech.org/archive/Interspeech_2018/abstracts/1070.html"
}Documents