MT-DNN

MT-DNN is Microsoft’s open-source PyTorch implementation of Multi-Task Deep Neural Networks for natural language understanding, with 2,200+ stars on GitHub — the official code behind the ACL 2019 paper of the same name.

Overview

MT-DNN starts from a simple observation: NLU tasks such as natural language inference, sentiment classification, semantic similarity, named entity recognition and machine reading comprehension share a lot of underlying language knowledge, so training them together should beat training them alone. The framework combines a shared transformer encoder — BERT in the original work — with task-specific output layers, pretrained on language modeling and then fine-tuned jointly across GLUE-style tasks, with knowledge distillation variants that improve the student further. The repository, last updated in 2024, also includes the team’s follow-on work: ALUM adversarial training for language model pretraining and fine-tuning, and a hybrid neural network model for commonsense reasoning. It remains a standard reference implementation for anyone studying multi-task learning for NLU or reproducing the paper’s results, with the architecture laid out as readable research code rather than a packaged product.

Key Features

• Official implementation – code for the ACL 2019 MT-DNN paper by Microsoft researchers.
• Multi-task fine-tuning – joint training across NLI, classification, ranking, NER and reading comprehension.
• Knowledge distillation – improved NLU via the distillation method from the companion paper.
• Adversarial training – ALUM code for large-scale adversarial LM pretraining and fine-tuning.
• Commonsense reasoning – hybrid neural network model code included in the v0.2 tree.
• PyTorch based – readable, hackable research code under the MIT license.

💳 Pricing

MT-DNN is free and open source under the MIT license. It is research code you run on your own hardware; there is no hosted service, paid tier or commercial offering attached to it.

Who Is It For

NLP researchers and engineers studying multi-task learning, reproducing classic NLU results, or looking for a proven starting architecture for joint training across language understanding tasks.