Efficient Inference for Dynamic Topic Modeling with Large Vocabularies

Abstract

Dynamic topic modeling is a well established tool for capturing the temporal dynamics of the topics of a corpus. Currently, dynamic topic models can only consider a small set of frequent words because of their computational complexity and insufficient data for less frequent words. In this work, we de- velop a scalable dynamic topic model by utilizing the correlation among the words in the vocabulary. By correlating previously independent temporal processes for words, our new model allows us to reliably estimate the topic representations contain- ing less frequent words. We develop an amortised variational inference method with self-normalised importance sampling approximation to the word distribution that dramatically reduces the compu- tational complexity and the number of variational parameters in order to handle large vocabularies. With extensive experiments on text datasets, we show that our method significantly outperforms the previous works by modeling word correlations, and it is able to handle real world data with a large vocabulary (80K words) which could not be pro- cessed by previous continuous dynamic topic mod- els. With qualitative analyses, we show that our method can perform inference on infrequent but representative keywords much more reliably than previous methods.

Related

April 2025 | 2024 IEEE Spoken Language Technology Workshop (SLT)

Classification Of Spontaneous And Scripted Speech For Multilingual Audio

Shahar Elisha, Andrew McDowell, Mariano Beguerisse-Díaz, Emmanouil Benetos

October 2024 | CIKM

PODTILE: Facilitating Podcast Episode Browsing with Auto-generated Chapters

A. Ghazimatin, E. Garmash, G. Penha, K. Sheets, M. Achenbach, O. Semerci, R. Galvez, M. Tannenberg, S. Mantravadi, D. Narayanan, O. Kalaydzhyan, D. Cole, B. Carterette, A. Clifton, P. N. Bennett, C. Hauff, M. Lalmas-Roelleke

October 2024 | Journal of Online Trust & Safety

Algorithmic Impact Assessments at Scale: Practitioners’ Challenges and Needs

Amar Ashar, Karim Ginena, Maria Cipollone, Renata Barreto, Henriette Cramer