Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction

Abstract : This paper introduces the Sequential Monte Carlo Transformer, an original approach that naturally captures the observations distribution in a transformer architecture. The keys, queries, values and attention vectors of the network are considered as the unobserved stochastic states of its hidden structure. This generative model is such that at each time step the received observation is a random function of its past states in a given attention window. In this general state-space setting, we use Sequential Monte Carlo methods to approximate the posterior distributions of the states given the observations, and to estimate the gradient of the log-likelihood. We hence propose a generative model giving a predictive distribution, instead of a single-point estimate.
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-02896961
Contributor : Sylvain Le Corff <>
Submitted on : Saturday, December 12, 2020 - 1:39:27 PM
Last modification on : Thursday, July 1, 2021 - 5:16:03 PM

Files

smc_transformer_2020.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02896961, version 2
  • ARXIV : 2007.08620

Citation

Alice Martin, Charles Ollion, Florian Strub, Sylvain Le Corff, Olivier Pietquin. The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction. 2020. ⟨hal-02896961v2⟩

Share

Metrics

Record views

44

Files downloads

354