GeezerVison

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages) An editor has determined that sufficient sources exist to establish the subject's notability. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "XLNet" – news · newspapers · books · scholar · JSTOR (September 2024) (Learn how and when to remove this message) This article may contain original research. Please improve it by verifying the claims made and adding inline citations. Statements consisting only of original research should be removed. (September 2024) (Learn how and when to remove this message) This article needs more citations. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "XLNet" – news · newspapers · books · scholar · JSTOR (September 2024) (Learn how and when to remove this message) (Learn how and when to remove this message)

The XLNet was an autoregressive Transformer designed as an improvement over BERT, with 340M parameters and trained on 33 billion words. It was released on 19 June 2019, under the Apache 2.0 license.^[1] It achieved state-of-the-art results on a variety of natural language processing tasks, including language modeling, question answering, and natural language inference.

The main idea of XLNet is to model language autoregressively like the GPT models, but allow for all possible permutations of a sentence.^[2] Concretely, consider the following sentence:

My dog is cute.

In standard autoregressive language modeling, the model would be tasked with predicting the probability of each word, conditioned on the previous words as its context:

We factorize the joint probability of a sequence of words $x_1,\dots ,x_T$ using the chain rule:$Pr(x_1,\dots ,x_T)=Pr\left(x_1\right)Pr\left(x_2\right|x_1)Pr(x_3|x_1,x_2)\dots Pr\left(x_T\right|x_1,\dots ,x_{T-1}).$

For example, the sentence "My dog is cute" is factorized as:

$Pr(\text{My},\text{dog},\text{is},\text{cute})=Pr\left(\text{My}\right)Pr\left(\text{dog}\right|\text{My})Pr(\text{is}|\text{My},\text{dog})Pr\left(\text{cute}\right|\text{My},\text{dog},\text{is}).$

Schematically, we can write it as

$\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\to \text{My }\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\to \text{My dog }\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\to \text{My dog is }\text{<𝙼𝙰𝚂𝙺>}\to \text{My dog is cute}.$

However, for XLNet, the model is required to predict the words in a randomly generated order. Suppose we have sampled a randomly generated order 3241, then schematically, the model is required to perform the following prediction task:

$\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\to \text{<𝙼𝙰𝚂𝙺>}\text{<𝙼𝙰𝚂𝙺>}\text{is }\text{<𝙼𝙰𝚂𝙺>}\to \text{<𝙼𝙰𝚂𝙺>}\text{dog is }\text{<𝙼𝙰𝚂𝙺>}\to \text{<𝙼𝙰𝚂𝙺>}\text{dog is cute}\to \text{My dog is cute}$

By considering all permutations, XLNet is able to capture longer-range dependencies and better model the bidirectional context of words.

Two models were released:^[1][2]

• XLNet-Large, cased: 340M parameters, 24-layer, 1024-hidden, 16-heads
• XLNet-Base, cased: 110M parameters, 12-layer, 768-hidden, 12-heads.

It was trained on a dataset that amounted to 32.89 billion tokens after tokenization with SentencePiece. The dataset was composed of BooksCorpus, and English Wikipedia, Giga5, ClueWeb 2012-B, and Common Crawl.

It was trained on 512 TPU v3 chips, for 5.5 days. At the end of training, it still under-fitted the data, meaning it could have achieved lower loss with more training. It took 0.5 million steps with an Adam optimizer, linear learning rate decay, and a batch size of 8192.^[3]

• BERT (language model)
• Transformer (machine learning model)
• Generative pre-trained transformer
• Yang Zhilin