A Systematic Review of Transformers

Abstract

In our world today, the way we live and interact has continued to experience rapid development. This feat has largely been enabled by Artificial Intelligence (AI), machine learning, and Neural networks. The advent of transformer model in deep learning has proven to be very revolutionary. This is largely traceable to the self-attention mechanism it adopts. Unnoticeably, we interact with transformer easily today for example, Google uses BERT to enhance its search engine by better understanding users’ search queries. Equally, it has been repeatedly mentioned on different media platform that transformers of the GPT family from openAI because of its capability to generate human-like characters and images. These successes and many others have attracted plenty to interest from academic researchers and the industry. In this study, the basic architecture of a transformer was examined including various literatures that surveyed transformers. This study showed the application of transformers in the machine translation, document summarization, document generation, named entity recognition, biological sequence analysis, market intelligence, character recognition. This study also proffers some solutions to some of the challenges with transformers