Language modeling, Machine translation CS 4803 / 7643 Deep Learning - - PowerPoint PPT Presentation

Language modeling, Machine translation

CS 4803 / 7643 Deep Learning
 Georgia Tech, 31 March 2020 James Cross, Jean Maillard Facebook AI

(

• ) = (, , . . . , )

= () ( | ) ( | , ) · · · ( | −, . . . , )

= Y

• ( | −, . . . , )

next word history

Language modelling aims to assign probabilities to word sequences

Search

Search Engine

how do I stop my backpack from ruining ruining my life ruining my relationship ruining my meal ruining your day ruining my hair ruining my carpet ruining your hair ruining my garden

( | −) = (, −) (−) ' (, −) (−)

(

• ) =

Y

• ( | −, . . . , )

= Y

• ( | −)

How many times “wi-1” appears … … in a large body of text, like Wikipedia. How many times “wi wi-1” appears … Bigram language model

= () ( | ) ( | , ) ( | , , ) · · ·

conditional independence (Markov assumption) ' () ( | ) ( | , ) ( | , , ) · · ·

RNN cell

• <latexit sha1_base64="rGpfCw03yA1iTferFVMd3+82y28=">ACSnicjZDNThsxFIU94S+EQgldmM1QqJSFM0gKrYRIMoCVFoIRGKi6I5zk1ix5btBNBoHoEtvFRfgNdgV3VT52fRwIa7OvLxd+7VSbTg1oXhc1BYWFxaXimultY+rG983CxvXVk1NAwbTAlmglYFDzFhuNOYFMbBJkIvE4Gh2P/eoTGcpVeunuNLQm9lHc5A+efLm7bUXuzEtbCydC3IpqJCpnNebscRHFHsaHE1DEB1t5EoXatDIzjTGBeKsVDixrYAHp40xlxbVOQaFvZ3eTe/H8/A2kluH41kR7cpk08hUts0lvu+vQYGCZKDegvhA6amvcbFmsZLd5bSrDE2U6FDuQHA2FzvOtBqZX2bR20ICT7sqdXbniPe4s9VT30Ba/W4QB1+zV5um1CRjzLwPKcVH6PsweOa5HxoNOGWyWOeZzn3J0etK34qr3Vq0V/v2c69SP5jVXSfyReyQyKyT+rkhJyTBmGkRx7I3kKfgcvwZ/g7/RrIZgxn8jcFBb/AaY4rzM=</latexit>

RNN cell

• <latexit sha1_base64="dBYkVpPm536KmLosBInGLrJeQA=">ACSnicjZDNbhMxFIU9IaUlUEhycYiQipSFM1URd1GbQRdBEgaSNlouiOc5NasceW7fyg0TxCt+1L8QK8BjvEBudnQZoNd3Xk4+/cq5Nowa0Lw59B4VFx7/H+wZPS02eHz1+Uj15eWTU1DtMCW6CVgUPMWO405gVxsEmQi8TiYXS/96hsZylbd419CeOUjzgD5+zQcng3IlrIWrobsi2ogK2UxrcBRE8VCxqcTUMQHW9qJQu34GxnEmMC+V4qlFDWwCY+wNZ1zbFCTafrZY3Zv/62cgrQR3U02kB9/SLjahjV065+6GfgCGiVIT+hVhiKbm/Y5FGmuZIucjpShiRJDyh0IzrZil5lWI/PLHpbSODpSKXOHjf4mDtbfoG0upHgzh5lz3YtKZWGUvm/5BS3EDfh8FPnvus0YBTJot1nunclxw9rHRXJ3UotPa+y+nlfr5pu4D8pq8IckImekTi5Ji3QI2NyS+7IfAj+BX8Dv6svxaCDfOKbE2h+BeoHK80</latexit>

RNN cell

• <latexit sha1_base64="N7oW9JQqfcVeOF0u6dk8ufHIgE0=">ACSnicjZDNThsxFIU9IW0htOVvycYiQkqlKJpQd1GgFoWQ0tCZGYKLrj3AQr9tiynUA1mkdgW16KF+A12CE2dX4WBTa9qyMf+denUQLbl0Y3geFpeKbt+WV0qr7z98XFvf2GxbNTYMW0wJZToJWBQ8xZbjTmBHGwSZCDxPRodT/3yCxnKVnrnfGrsShikfcAbOP/26n3prZfDWjgb+lpEC1Emi2n2NoIo7is2lpg6JsDaiyjUrpuBcZwJzEuleGxRAxvBEC/6E65tChJtN7ue3Zv/62cgrQR3WU2kB3dpBxtwh16xd0l/QYME6VG9CdCH03N+y2LNYya5zOlCGJkr0KXcgOHsWO820GplfZtHbQgJPByp1tnLEh9zZasM3kFa/G8TRp+zFpjk1y5gy/4eU4iP0fRg8dwPjQacMlms80znvuToZaWvRftzLdqr7Z/ulesHi7qXyTbZIRUSka+kTo5Jk7QI0NyQ/6Q2+AueAgeg6f510KwYLbIsykU/wKqAK81</latexit>

RNN cell

• <latexit sha1_base64="Vno21VB0DMdmb10OrfE2uFj/e8k=">ACSnicjZDNbhMxFIU9oS1toCWBJRuLClIUTRTBbGNIugi0aSNlouiOc5NYsceW7TRBo3mEbtuX4gX6GuwQG5yfBWk3vasjH3/nXp1EC25dGN4FhSc7u3tP9w+Kz54fHr0olV9eWDUzDtMCW6CVgUPMWO405gVxsEmQi8TKaflv7lFRrLVXrufmrsSxinfMQZOP90Nh80BqVKWA9XQx+KaCMqZDOng3IQxUPFZhJTxwRY24tC7foZGMeZwLxYjGcWNbApjLE3vOLapiDR9rPF6t78fz8DaSW4S2RHnxLu9iGc+zSOXcT+hkYJkpN6Q+EIZq69zsWaxlki1yOlKGJkoMKXcgONuKXWZajcwvs+htIYGnI5U6W23xMXe21vYNpLUvBnH6Lru3aU2tMpbM45Bi3ELfh8Gvnvum0YBTJot1nunclxzdr/ShuDiuR436+NSvPjpu598pq8IVUSkQ+kSU7IKekQRsbkmtyQ2+BX8Dv4E/xdfy0EG+YV2ZrCzj+r5K82</latexit>

(|, , , )

RNN language models don’t need a Markov assumption. The more context we have to predict the next word, the better!

Inference Given a trained RNN language model:

• Feed the words one by one, starting with a special start symbol <s>.

Stop when the special end symbol </s> is predicted. At time step t+1:

• Feed the embedding corresponding to the word predicted at t.

At every time step t, to predict:

• Project the RNN output to the vocabulary space.
• Normalise using softmax to obtain a valid distribution.
• Sample from this distribution to predict the next word.

…

• <latexit sha1_base64="74qbGxPHZlSmXlqoyuImvaZ8tgk=">ACTnicjZBLSwMxFIUz9VXrW5dugkVQKGVGFLeioi4UH1hbcIrcSW/bMkJKkPhvkRbvVPufWPuBOd1i58bLyrQ06+cy8n0oJb5/uvXmFkdGx8ojhZmpqemZ2bX1i8sqpnGNaYEso0IrAoeI1x53AhjYIMhJYj+K9vl+/RWO5Si7dg8amhE7C25yBy5/qoZR2stu5st+1R8M/SuCoSiT4ZzdLHhB2FKsJzFxTIC14GvXTMF4zgTmJVKYc+iBhZDB69bt1zbBCTaZno/uDn7qcgrQTXrUQyB1dpA4/hEhv0jrsuPQCGkVIxvUBoanmfs0iHRx+n9G2MjRSokW5A8HZj9h+ptXI8mUWc1tI4ElbJc6u7fMOd7ZynLeQVA4NYrye/tr0RQ0y+sz/kFK4j3kfBk9y7lSjAadMGuos1f2Sg9+V/hVXG9Vgs7p1vlne2R3WXSTLZIWskYBskx1yRM5IjTASk0fyRJ69F+/Ne/c+vr4WvCGzRH5MofgJDOxXA=</latexit>

RNN cell People RNN cell the RNN cell embedding size RNN cell We vocabulary size User input: P e

• p

l e c

• m

m

• n

a r e y e a r n e w a s

• m

e t h i n g t h e i s w i t h a n d s e x t a n t m a n g

• Next word prediction:

Inference: finding p(w | “We the”)

RNN cell <s> = (

• ) =
• P

projection

How do we know how well our RNN models language? If our model is good, it will assign high probability to real sentences.

(∗, ) = − X

∈X

∗() ()

cross entropy reference distribution expected # bits required to represent an event drawn from p*, when using a coding scheme optimal for p.

= −

• X
• ( | −, . . .)

per-word cross entropy commonly used as loss function

How do we know how well our RNN models language? Perplexity is the geometric mean of the inverse probability of a sequence of words (according to the model).

(

• ) =
• Y
• s
• ( | −, . . .)

Note:

• The exponent is our loss function, the per-word cross entropy!
• The lower the perplexity, the better our model is.

= −

• P

(|−,...)

commonly used as evaluation metric

Training Given a long list of words:

• Feed them one by one, enclosing every sentence with <s> and </s>.

Minimisation of this loss is equivalent to maximum likelihood estimation. At every time step t, compute the predictive distribution pt.

• Compute ℒ t, the loss at time step t, as the cross entropy between the

predictive distribution and the actual next word.

• The total loss ℒ is the average of the loss at every time step.

. . .

RNN cell 24th

• <latexit sha1_base64="zubXSPfzgz+uqeTaeUkeuyFGm4M=">ACdXicjVHNbhMxGHSWv5IWSOGIkCxaIJGiZbdp2uRWQVU4tGqLmnalboi8zreJFXt2U4JWu1L8DS9wlvwJEg94U1aiSIhMaeRxzOfv3GiODM2CH5WvDt3791/sPSwurzy6PGT2urTUyOnmkKPSi51lBADnGXQs8xyiJQGIhIOZ8nkfamfXYA2TGYn9quCviCjKWMEuOBrVmLBVoYqXOiIDcyNQKMitiDqmtx0ok+bT4vBlrNhrbxqC2FvjdzlY7bGFHtjtb3XZJuhutTguHfjDH2k6jdRxdvrk6GqxWwngo6VRAZiknxpyHgbL9nGjLKIeiWo2nBhShEzKC8+EFU6Z8huns/lqxZ96ToQRxI6biXDGVziCfXICEf7C7BjvEQqJlBP8CcgQtO/0ngE832BW4FRqnEg+xMwSzuit2DLTKBumAEnc0FYlsrMmvouGzFrmvurKz5QNMGvlfkxaueUbp+T9LNd4F14eGA+c7vP6APFZFrgpX8k2T+N/kdMPN/32sWv7HVpgCT1HL1EdhWgb7aCP6Aj1EXf0CX6jn5UfnkvHXv9eKqV7n2PEO34L39DSkjwlE=</latexit>

L

RNN cell the

• <latexit sha1_base64="IXJ1b+L8BYDcpl2qzhrBM0b0FG4=">ACdXicjVHdbtMwGHXD31b+OrhESBbjp5WqkCx0a+8mNgEXQ2xo3SItXeW4X1qrdmzZ7lYU5SV4mt2Ot+BJkLjCaYfEkJA4V0c+Pufzd5wqzowNgu8178bNW7fvrKzW7967/+BhY+3RkZEzTaFPJZc6TokBznLoW2Y5xEoDESmH43S6U+nHZ6ANk/mh/aJgIMg4ZxmjxLqjYaOdSAWaWKlzIqAwMrOCzMuEQ2abiRJpMStPo0Sz8cS2ho31wO91NzthB3Z6m72OhXpbUTdCId+sMD6dis6iC9e/dwfrtXCZCTpTEBuKSfGnISBsoOCaMsoh7JeT2YGFKFTMoaT0RlTpnqGRTzxWrln3pBhBHETtqpcMYXOIY9cgxPmd2gt8RCqmU/wZyAi07/S+AbzYF7iTGqcSj7CzBLO6LXYKtMoG6YASdzQVieydya5i4bM2vae6svP1eA0xbxV+Tlq5FRuX5P0s92QXh4aPzvfp6gOKRJWFKl3Jv5vE/yZHG374xu8cuLbfoiVW0BP0DVRiLbQNvqA9lEfUfQVXaBL9K32w3vqPfdeLq96tSvPY3QN3utfJzjCUA=</latexit>

L

RNN cell On

• <latexit sha1_base64="ie8F9gy2q4Zza84lvK0Ca3fcyeA=">ACdXicjVHbhMxFHS2XEq4peURVbIol0SKlt2EtMlbRSvgoYgWNe1K3TyOmcTK/basp2SarU/wdf0tfwFX4LE96kSBQJiXkaeTxzfMaJ4szYIPhe8VZu3b5zd/Ve9f6Dh48e19bWj42caQp9KrnUIMcJZB3zLIVIaiEg4nCT3VI/OQdtmMyO7IWCgSDjKWMEuOhrVmLBVoYqXOiIDcyNQKMi9iDqmtx0ok+aw4a8WajSe2MaxtBn6vu9UJ29iR7e5Wr1OSXqvdbePQDxbY3Gm0D6PLVz8PhmuVMB5JOhOQWcqJMadhoOwgJ9oyqGoVuOZAUXolIzhdHTOlCmfYQb5fLFa8aeE2EsZNmIpzxBY5gnxBhL8wO8HvCIVEyin+DGQE2nd63wBebDAvcCo1TiQfYWYJZ/RGbJlpFA3zICTuSAsS2VmTX2PjZk1zX1XVtZ8rwGmjfyvSUvXIqP0/J+lGu+B60PDR+f7dP0BeayKXBWu5N9N4n+T45YfvE7h67t2iJVfQUPUN1FKJtIM+oAPURxR9RZfoCn2r/PA2vOfey+Vr3LteYJuwHv9CyVNwk8=</latexit>

L

Training, given the sentence “On the 24th of February 1810, …”

+ + + =

• X
• L

+

. . .

L =

L =

• ,

= −

+

L = −

• <latexit sha1_base64="IntfmxfORVYbSgneI49jLfZgYdE=">ACeXicjVDLbhMxFHWmQEt4pbBk41IhBRGMxSEVISoAIWQS2oaSPVaeRxbqZW/JLtlCBrfoMfYcsWVvwAH8BXsMGTdEHbDWd15HPudenMI7n2W/GsnSpctXleuNq9dv3HzVmv19r7TU8ugx7TQtl9QB4Ir6HnuBfSNBSoLAQfF5HWtH5yAdVyrPf/ZwEDSUvExZ9THp2ErIx5mPkBaphV5TiT1x4yK0K2OcvwCPyJCl9gc5cPF2I6qhq31LM3mwBdJfkrWt9Z+fv093Xy5O1xt5GSk2VSC8kxQ5w7zPhBoNZzJqBqNsnUgaFsQks4HJ1w4xSV4AZhNv9e9a8eqHT1jZ1CRuN93Icu3YM+/sT9MX5DGRaT/BHoCOwadR7DjAxsgizCo+1xYUWI8w9FZydia0znQEWlzmIspCUq7FW3rW3ecm963RjYarz1gJMHoRzmxaueUbt+T9Lk2xD7MPC+jbMWCp1zYQUwVTl5yfr/Qi2X+c5k/Spx9i26/QAivoLrqH2ihHz9AWeod2UQ8x9AV9Q9/Rj8afZC1pJw8Xo0nj1HMHnUGy8RdzTcJO</latexit>
• <latexit sha1_base64="4pvnAvo3qYjFV7k5BVXWHqD/6Ns=">ACUHicjVBNTxsxFHyblgJbaIEeuViNKoEURbuoiCsCRHsAlVYEgtiA3jovwYq9tmyHglb7K3pt/xQ3/gm31gmpxMeFuXj0xjPvaXIjhfNJchvVXr2ejM9Mxu/nZt/935hcenI6aHl1OJatvO0ZEUBbW8JLaxhKqXNJxPtge6ceXZJ3QxaG/NtR2C9ET3D0YXSGZWXw+osPV+oJ81kDPacpBNShwkOzhejNOtqPlRUeC7RudM0Mb5TovWCS6riOBs6MsgH2KfT7qUwrkBFrlNeja+uHuolKqfQXzRyFYyfWJv28JDa7KfwF2wXOeVaD9gPwi7ZtBbjtj49KuK9bRluZdJjxKwR/FjKdIR6WOQqyVCiKni68W9kRfeFdYy/0UDS+WKLBavlk071rnDHyvMwSZzsU+rC0H3zfDFn02paZqUpThZLTp5U+J0drzfRzc/17Ut/cmtQ9A8vwEVYghQ3YhK9wAC3goOAX/IY/0U10F/2tRfdf/7/wAR6hFv8DSkCw/A=</latexit>

RNN cell <start>

• <latexit sha1_base64="X1E+ASFqN/HGMO9swgkm8IfeDMY=">ACdXicjVHNbhMxGHSWAiX8pfRYVbIoP4kULbsNaZNbVSrgUESLmnalbhp5nW8TK/basp2SarUv0afplb4FT4LECW9SJIqExJxGHs98/saJ4szYIPhe8e4s3b13f/lB9eGjx0+e1laeHRs51R6VHKpo4QY4CyDnmWQ6Q0EJFwOEkm70r95By0YTI7shcK+oKMpYySqw7GtSasVSgiZU6IwJyI1MryKyIOaS2HiuR5NPiLIw1G41tY1DbCPxuZ6sdtrAj252tbrsk3c1Wp4VDP5hjY6fROoyuXv8GKxUwngo6VRAZiknxpyGgbL9nGjLKIeiWo2nBhShEzKC0+E5U6Z8huns/lqxZ96ToQRxI6biXDGlziCfXIEf7K7Bi/JxQSKSf4C5AhaN/pPQN4vsGswKnUOJF8iJklnNFbsWmUDdMANO5oKwLJWZNfU9NmLWNPdWVnzgwaYNPK/Ji1c84zS83+WarwHrg8Nn5zv80H5LEqclW4kn83if9Njf98K3fPnRt76IFltEaeo7qKETbaAd9RAeohyi6RFfoG7qu/PDWvRfeq8Vr3LjWUW34L35BSNiwk4=</latexit>

L

• <latexit sha1_base64="Cj/Rsa6rgAEt5HFJ8SuSQv0jP0s=">ACUHicjZBNTxRBEIZrFhUYVECPeOhITDZbGYIxCtRghwomFhDbOSmt7a3c52T3e6e/nIZH4FV/hT3Pgn3rR3VhM+LtalK/X2+1blyY0UzifJbdSYefL02ezcfLzw/MXLxaXlV4dOjy2nNtdS206OjqQoqO2Fl9QxlDlko7y0aeJfnRK1gldHPgLQ12Fg0L0BUcfRj8yo/LSVD/Tk6XVpJXUxR436d9mdesN1LV/shylWU/zsaLCc4nOHaeJ8d0SrRdcUhXH2diRQT7CAR3ToVxBSpy3fK8vrq6q5eonEI/bOYqGN+xDu3hAXYmfBDtoOcq1H7Dthj2wr6G1HrD79vGJ9bVmuZY8Jj1Lwe7GTGeIh2WOgiwViqKvC+/WtsVAeNfcCxyK5mdLNHpfPtg0dUZE8/WeJsmwIPS1+C76shi17bMjNVwBwgpw+RPm4O1vpRmvzW6D9cUob5mAF3sIapPABtmAX9qENHBRcwhVcRzfRr+h3I5p+/fCa7hXjfgP4DGxUw=</latexit>

Applications

• Predictive typing:
• In search fields
• For keyboards
• For assisted typing, e.g. sentence completion
• Automatic speech recognition:
• How likely is the user to have said My hair is wet vs My hairy sweat?
• Basic grammar correction:
• p(They’re happy together) > p(Their happy together)

Conditional language models Like a language model, but conditioned on extra context c.

(

• | ) =

Y

• ( | , −, . . . , )

• Sentiment analysis: c = {positive, negative, neutral}, s = text.
• Text summarisation: c = a long document, s = its summary.
• Machine translation: c = French text, s = English text.
• Image captioning: c = an image, s = its caption.
• Optical character recognition: c = image of a line, s = its content.
• Speech recognition: c = a recording, s = its content.

Masked language models

position embeddings word embeddings

<mask> <cls> a seat <mask> have a <mask> <s> 1 2 3 4 5 6 7 8

+ + + + + + + + +

transformer encoder

take </s> drink

predictions

Masked language models

• Don’t have to limit ourselves to just English. XLM-R is a single model

that works on 100 languages:

allons enfants de la patrie , le jour de gloire

• ! say , can you see , by the dawn ’s early

ee mungu nguvu yetu Ilete baraka kwetu

• जन-गण-मन अिधनायक जय हे, भारत भाग्र िवधाता! पंजाब

… … … … …

• No need to explicitly tag the input language.
• For many cases, it performs better than language-specific models.
• Achieve state-of-the-art on a variety of tasks.

Model size in perspective

performance (GLUE) 50 63 75 88 100 number of parameters (millions) 10 100 1000 10000 100000 CBoW (2013) BERT (2018) RoBERTa (2019) T5 (2019) ELMo (2018) embedding-based RNN transformer Types of model: human baseline

Knowledge distillation Idea:

• Take a big model, use it to teach a smaller (and therefore faster) model.

These models work very well, but they are often too slow!

• Can we make the smaller model work as well as the larger one?

Can we make them faster without affecting their NLP performance?

Input text Model Prediction Target training loss

standard training

Knowledge distillation

Input text Model Prediction Target training loss

standard training

distillation loss Input text Pretrained teacher model Student model Prediction Prediction

knowledge distillation, first attempt

Knowledge distillation

Input text Model Prediction Target training loss

standard training

distillation loss Input text Pretrained teacher model Student model Prediction Prediction

knowledge distillation, first attempt

distillation loss student loss Input text Target Pretrained teacher model Student model Prediction Prediction

knowledge distillation, second attempt

Knowledge distillation

Input text Model Prediction Target training loss

standard training

distillation loss Input text Pretrained teacher model Student model Prediction Prediction

knowledge distillation, first attempt

distillation loss student loss Input text Target Pretrained teacher model Student model Prediction Prediction

knowledge distillation, second attempt

distillation loss student loss Input text Target Pretrained teacher model Student model Soft predictions Soft predictions

knowledge distillation

Knowledge distillation

References

• Bengio et al. (2003). A Neural Probabilistic Language Model. In JMLR.
• Sutskever et al. (2014) Sequence to sequence learning with neural networks. In NIPS 14.
• Devlin et al. (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language
• Understanding. In NAACL 2019.
• Liu, Ott, Goyal, Du et al. (2019) RoBERTa: A Robustly Optimized BERT Pretraining
• Approach. arXiv preprint 1907.11692.
• Conneau, Khandelwal et al. (2019) Unsupervised Cross-lingual Representation Learning

at Scale. arXiv preprint 1911.02116.

• Lewis et al. (2019) BART: Denoising Sequence-to-Sequence Pre-training for Natural

Language Generation, Translation, and Comprehension. arXiv preprint 1910.13461.

• Raffel et al. (2019) Exploring the Limits of Transfer Learning with a Unified Text-to-Text
• Transformer. arXiv preprint 1910.10683.

Neural Machine Translation

Languages are different

• I saw a movie

Translation is a Conditional Language Model

mon sac à dos ruine ma vie ! my backpack is ruining my _____

Sequence-to-Sequence: Naive RNN

x1 x2 x3 x4 θ y1 y2 y3 y1 y2 y3 <end>

Sequence-to-Sequence: Attention

x1 x2 x3 x4 θ y1 y2 y3 y1 y2 y3 <end> encoder outputs

Attention as Alignment

I saw a ___

p = 0.9 p = 0.08 p = 0.02

Transformer NMT

figure from Vaswani et al. 2017

• NMT is original application of

transformer architecture

• Encoder-decoder attention

considers all encoder outputs

• Decoder self-attention considers
• nly previous outputs
• Inference is autoregressive (left-

to-right) but decoder can be trained in parallel

Beam Search

• I

_EOS watched saw watch saw watched the a a movie the a movie movie movie _EOS _EOS _EOS _EOS

Beam Search

• Each beam element has different state (for transformer decoder,

this is self-attention input for previous steps)

• Total computation scales linearly with beam width
• However computation is highly parallelizable over the beam

(efficient on GPU)

• Inference is autoregressive (each step depends on previous

decision)

Non-Autoregressive Machine Translation (NAT)

figure from Gu et al. 2017

Variants of Non-Autoregressive Translation (NAT)

• Original approach: Gu et al. 2018
• Fertility Prediction
• Noisy Parallel Decoding
• Conditional Masked Language Models: Ghazvininejad et. al 2019
• Training with randomly masked target tokens
• Inference: iteratively predict highest-probability target tokens
• Levenshtein Transformer: Gu et al. 2019
• Separate decoders for insertion, prediction, and deletion
• Trained with reinforcement learning

• Problem: how to model rare or

unseen words?

• Character-level models are

general but too slow in practice

• Subword models solve break up

words based on frequency

• One widely used algorithm is

Byte-Pair Encoding (BPE), illustrated at right Byte-Pair Encoding (BPE)

example from Sennrich et al. 2016

• Problem: how to model rare or

unseen words?

• Character-level models are

general but too slow in practice

• Subword models solve break up

words based on frequency

• One widely used algorithm is

Byte-Pair Encoding (BPE), illustrated at right Byte-Pair Encoding (BPE) token frequency

l

• w

e r

</w>

2 l

• w

</w>

5 n e w e s

</w>

6 t w i d e s

</w>

3 t

learned merge operations

r • → r• l o → lo lo w → low e r• → er•

example from Sennrich et al. 2016

Ongoing challenges for machines translation

• Low-resource language directions
• Massively multilingual models
• Iterative back-translation
• Data mining
• Noisy input text (such as internet slang)
• Character-aware translation
• Adversarial training techniques
• Catastrophic failures
• Improved quality estimation, identification techniques

References

• Bahdanau et al. Neural Machine Translation by Jointly Learning to Align and Translate

(2014)

• Ghazvininejad et al. Mask-Predict: Parallel Decoding of Conditional Masked Language

Models (2019)

• Gu et al. Non-Autoregressive Neural Machine Translation (2017)
• Gu et al. Levenshtein Transformer (2019)
• Ott et al. Scaling Neural Machine Translation (2018)
• Sennrich et al. Neural Machine Translation of Rare Words with Subword Units (2016)
• Sutskever et al. Sequence to Sequence Learning with Neural Networks (2014)
• Vaswani et al. Attention is All You Need (2017)