Paper Image

Joint language modeling of speech and text

Published on:

12 October 2023

Primary Category:

Computation and Language

Paper Authors:

Ju-Chieh Chou,

Chung-Ming Chien,

Wei-Ning Hsu,

Karen Livescu,

Arun Babu,

Alexis Conneau,

Alexei Baevski,

Michael Auli


Key Details

Compares speech tokenization methods like HuBERT and SentencePiece

Introduces techniques to mix speech and text data

Joint model transfers better between modalities

Proposes new automatic evaluation metrics

Alternating speech & text helps learn shared representations

AI generated summary

Joint language modeling of speech and text

This paper explores joint language modeling of speech units and text. It compares speech tokenization methods and techniques for mixing speech and text data. The joint model shows improved ability to transfer between modalities on downstream tasks. New automatic evaluation metrics are also proposed.

Answers from this paper


No comments yet, be the first to start the conversation...

Sign up to comment on this paper

Sign Up