Event
Generatives Vortraining für Sprachmodelle
In 2018 OpenAI researchers introduced an approach combining the Transformer architecture, generative pre-training on text without task labels and subsequent fine-tuning for specific language tasks. In 2019 OpenAI retrospectively called the approach GPT. Original paper and context
The researchers reported improvements over the previous state of the art on nine of twelve tasks studied; these are results on particular tests, not proof of general language understanding. OpenAI also noted brittle generalisation. Sources GPT-2 followed in 2019.
Sources
- Originalarbeit und OpenAI-Beiträge: Verfahren, Ergebnisse und spätere Bezeichnung.
Offen
Trainingsdaten und Vergleich zu gleichzeitigem BERT anhand weiterer Quellen vertiefen. / Examine training data and comparison with contemporary BERT using additional sources.