The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction

Alice Martin (1, 2, 3, 4) , Charles Ollion (1) , Florian Strub (5) , Sylvain Le Corff (2, 3, 4) , Olivier Pietquin (6)

1 CMAP - Centre de Mathématiques Appliquées - Ecole Polytechnique
2 IP Paris - Institut Polytechnique de Paris
3 TSP - CITI - Communications, Images et Traitement de l'Information
4 TIPIC-SAMOVAR - Traitement de l'Information Pour Images et Communications
5 DeepMind [London]
6 Google Brain, Paris

This paper introduces the Sequential Monte Carlo Transformer, an original approach that naturally captures the observations distribution in a transformer architecture. The keys, queries, values and attention vectors of the network are considered as the unobserved stochastic states of its hidden structure. This generative model is such that at each time step the received observation is a random function of its past states in a given attention window. In this general state-space setting, we use Sequential Monte Carlo methods to approximate the posterior distributions of the states given the observations, and to estimate the gradient of the log-likelihood. We hence propose a generative model giving a predictive distribution, instead of a single-point estimate.

Mots clés: Deep learning; Transformer; Monte Carlo methods; Sequence prediction

Domaines: Machine Learning [stat.ML]; Intelligence artificielle [cs.AI]; Applications [stat.AP]

Alice Martin, Charles Ollion, Florian Strub, Sylvain Le Corff, Olivier Pietquin. The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction. 2020. ⟨hal-02896961v2⟩