Thennal D K

I’m Thennal (any/all), a CS undergrad at the Indian Institute of Information Technology Kottayam, conducting natural language processing research since 2018. I’m interested in what makes language/speech models tick, and how to make them tick better. In particular:

How do large pretrained models form their internal representations, and how does each component update it?
There are a lot of pretrained and finetuned models available publicly. Can we use them to make better models?
The field has a significant evaluation/benchmarking problem, particularly when it comes to non-English languages. How can we make it better?

I also like running, fungi, anything produced by Supergiant Games, and Japanese music. Go watch Etsuko Yakushimaru’s I’m Humanity, and then read about it.

news

Jan 22, 2025	Our paper on ASR evaluation metrics was accepted to NAACL Findings 2025!
Oct 18, 2024	Two new preprints, related to my internship with the University of Hamburg and my collaboration Jesin James from the University of Auckland.
Feb 20, 2024	Paper accepted at LREC-COLING 2024! Excited to go there in May and present our work, Fisher Mask Nodes for Language Model Merging.
Feb 17, 2024	Got the DAAD WISE scholarship for an internship with the University of Hamburg!

latest posts

Oct 24, 2024	The lost art of checking your sources
May 01, 2024	Why do I chase the heat?
Dec 29, 2022	Whisper's evaluated metrics are kind of wrong for a bunch of languages

selected publications

Fisher Mask Nodes for Language Model Merging

Thennal D K , Ganesh Nathan , and Suchithra M S

In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024

Abs URL

Fine-tuning pre-trained models provides significant advantages in downstream performance. The ubiquitous nature of pre-trained models such as BERT and its derivatives in natural language processing has also led to a proliferation of task-specific fine-tuned models. As these models typically only perform one task well, additional training or ensembling is required in multi-task scenarios. The growing field of model merging provides a solution, dealing with the challenge of combining multiple task-specific models into a single multi-task model. In this study, we introduce a novel model merging method for Transformers, combining insights from previous work in Fisher-weighted averaging and the use of Fisher information in model pruning. Utilizing the Fisher information of mask nodes within the Transformer architecture, we devise a computationally efficient weighted-averaging scheme. Our method exhibits a regular and significant performance increase across various models in the BERT family, outperforming full-scale Fisher-weighted averaging in a fraction of the computational cost, with baseline performance improvements of up to +6.5 and a speedup between 57.4x and 321.7x across models. Our results prove the potential of our method in current multi-task learning environments and suggest its scalability and adaptability to new model architectures and learning scenarios.