Citation

Janis Pagel, Axel Pichler, Nils Reiter. Evaluating In-Context Learning for Computational Literary Studies: A Case Study Based on the Automatic Recognition of Knowledge Transfer in German Drama. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pp. 1-10, St. Julians, Malta, March 2024. Url: https://aclanthology.org/2024.latechclfl-1.1.

Downloads

Abstract

In this paper, we evaluate two different natural language processing (NLP) approaches to solve a paradigmatic task for computational literary studies (CLS): the recognition of knowledge transfer in literary texts. We focus on the question of how adequately large language models capture the transfer of knowledge about family relations in German drama texts when this transfer is treated as a classification or textual entailment task using in-context learning (ICL). We find that a 13 billion parameter LLAMA 2 model performs best on the former, while GPT-4 performs best on the latter task. However, all models achieve relatively low scores compared to standard NLP benchmark results, struggle from inconsistencies with small changes in prompts and are often not able to make simple inferences beyond the textual surface, which is why an unreflected generic use of ICL in the CLS seems still not advisable.

BibTeX

@inproceedings{pagel2024a,
   author    = {Janis Pagel and Axel Pichler and Nils Reiter},
   booktitle = {{Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024)}},
   location  = {St. Julians, Malta},
   month     = {3},
   pages     = {1--10},
   title     = {{Evaluating In-Context Learning for Computational Literary Studies: A Case Study Based on the Automatic Recognition of Knowledge Transfer in German Drama}},
   url       = {https://aclanthology.org/2024.latechclfl-1.1},
   year      = {2024},
}

RIS

TY - CPAPER
TI - Evaluating In-Context Learning for Computational Literary Studies: A Case Study Based on the Automatic Recognition of Knowledge Transfer in German Drama
AU - Janis Pagel
AU - Axel Pichler
AU - Nils Reiter
PY - 2024
J2 - Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024)
UR - https://aclanthology.org/2024.latechclfl-1.1
CY - St. Julians, Malta
SP - 1
EP - 10
ER -