ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

Maurya, K.K. and Desarkar, Maunendra Sankar and Kano, Y. and et al, . (2021) ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 1 August 2021 through 6 August 2021, Virtual, Online.

[img] Text
ACL_IJCNLP_2021.pdf - Published Version
Available under License Creative Commons Attribution.

Download (1MB)


Despite the recent advancement in NLP research, cross-lingual transfer for natural language generation is relatively understudied. In this work, we transfer supervision from high resource language (HRL) to multiple low-resource languages (LRLs) for natural language generation (NLG). We consider four NLG tasks (text summarization, question generation, news headline generation, and distractor generation) and three syntactically diverse languages, i.e., English, Hindi, and Japanese. We propose an unsupervised cross-lingual language generation framework (called ZmBART) that does not use any parallel or pseudo-parallel/back-translated data. In this framework, we further pre-train mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task using monolingual data of three languages. The objective function of the auxiliary task is close to the target tasks which enriches the multi-lingual latent representation of mBART and provides good initialization for target tasks. Then, this model is fine-tuned with task-specific supervised English data and directly evaluated with low-resource languages in the Zero-shot setting. To overcome catastrophic forgetting and spurious correlation issues, we applied freezing model component and data argumentation approaches respectively. This simple modeling approach gave us promising results. We experimented with few-shot training (with 1000 supervised data-points) which boosted the model performance further. We performed several ablations and cross-lingual transferability analysis to demonstrate the robustness of ZmBART. © 2021 Association for Computational Linguistics

[error in script]
IITH Creators:
IITH CreatorsORCiD
Desarkar, Maunendra Sankar
Item Type: Conference or Workshop Item (Paper)
Additional Information: We thank the support from Nvidia AI Technology Center (NVAITC) towards the requirements of computing power and compute infrastructure. We thank the human annotators for human evaluation and the anonymous reviewers for their constructive feedback.
Uncontrolled Keywords: Auto encoders; Cross-lingual; De-noising; Headline generation; Language generation; Low resource languages; Natural language generation; Objective functions; Pseudo parallels; Text Summarisation
Subjects: Computer science
Divisions: Department of Computer Science & Engineering
Depositing User: . LibTrainee 2021
Date Deposited: 07 Oct 2022 09:09
Last Modified: 07 Oct 2022 09:09
Publisher URL:
Related URLs:

Actions (login required)

View Item View Item
Statistics for RAIITH ePrint 10845 Statistics for this ePrint Item