moinsen / Software historyDE ↗
← 2018

Event

Generatives Vortraining für Sprachmodelle

Context and research status
  • Kennung: 2018-generatives-vortraining
  • Zeitraum: 11. Juni 2018
  • Datumsgenauigkeit: Tag / day
  • Art: Forschung / research
  • Kategorien: Sprachen und Techniken, Prägende Akteure, Ereignisse
  • Akteure und Rollen: Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever: Forschung; OpenAI: Veröffentlichung
  • Region: international

In 2018 OpenAI researchers introduced an approach combining the Transformer architecture, generative pre-training on text without task labels and subsequent fine-tuning for specific language tasks. In 2019 OpenAI retrospectively called the approach GPT. Original paper and context

The researchers reported improvements over the previous state of the art on nine of twelve tasks studied; these are results on particular tests, not proof of general language understanding. OpenAI also noted brittle generalisation. Sources GPT-2 followed in 2019.

Sources

Offen

Trainingsdaten und Vergleich zu gleichzeitigem BERT anhand weiterer Quellen vertiefen. / Examine training data and comparison with contemporary BERT using additional sources.