Publication detail

Speech and Language Recognition with Low-rank Adaptation of Pretrained Models

PRASAD, A. MADIKERI, S. KHALIL, D. MOTLÍČEK, P. SCHUEPBACH, C.

Original Title

Speech and Language Recognition with Low-rank Adaptation of Pretrained Models

Type

conference paper

Language

English

Original Abstract

Finetuning large pretrained models demands considerable computational resources, posing practical constraints. Major- ity of the total number of parameters in these models are used by fully connected layers. In this work, we consider applying a semi-orthogonal constraint, followed by full finetuning to the fully connected layers reduces model parameters significantly without sacrificing efficacy in downstream tasks. Specifically, we consider wav2vec2.0 XLS-R and Whisper models for Auto- matic Speech Recognition and Language Recognition. Our re- sults show that we can reduce the model size by approximately 24% during both training and inference time with 0.7% absolute drop in performance for XLS-R and no drop in performance for Whisper for ASR. In combination with performance-efficient training with low-rank adapters, the resource requirements for training can be further reduced by up to 90%.

Keywords

parameter reduction, language identification, speech recognition, wav2vec2.0

Authors

PRASAD, A.; MADIKERI, S.; KHALIL, D.; MOTLÍČEK, P.; SCHUEPBACH, C.

Released

1. 9. 2024

Publisher

International Speech Communication Association

Location

Kos Island

ISBN

1990-9772

Periodical

Proceedings of Interspeech

Year of study

2024

Number

9

State

French Republic

Pages from

2825

Pages to

2829

Pages count

5

URL

https://www.isca-archive.org/interspeech_2024/prasad24_interspeech.html

BibTex

@inproceedings{BUT193370,
  author="PRASAD, A. and MADIKERI, S. and KHALIL, D. and MOTLÍČEK, P. and SCHUEPBACH, C.",
  title="Speech and Language Recognition with Low-rank Adaptation of Pretrained Models",
  booktitle="Proceedings of Interspeech",
  year="2024",
  journal="Proceedings of Interspeech",
  volume="2024",
  number="9",
  pages="2825--2829",
  publisher="International Speech Communication Association",
  address="Kos Island",
  doi="10.21437/Interspeech.2024-2187",
  issn="1990-9772",
  url="https://www.isca-archive.org/interspeech_2024/prasad24_interspeech.html"
}

Documents

prasad_2024_interspeech.pdf