Citation

Axel Pichler, Janis Pagel. Evaluating Humanities Theory Alignment in Large Language Models: Incremental Prompting and Statistical Assessment. In Proceedings of the 10th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2026), pp. 280-294, Rabat, Morocco, March 2026. Url: https://aclanthology.org/2026.latechclfl-1.27.

Downloads

Abstract

We propose a method to evaluate the extent to which an LLM’s observable input–output behavior aligns with established theories in the humanities and cultural studies. We instantiate the framework on three humanities theories—Davidson’s truth-conditional semantics, Lewis’s truth in fiction, and Iser’s concept of textual gaps—using a top-down, theory-driven black-box framework. Core assumptions of these theories are reconstructed into testable behavioral rules and assessed via controlled classification tasks with systematic prompt comparisons and significance testing. Our experiments show that theory-uninformed classification prompts generally outperform theory-enriched prompts in Lewis and Iser settings, while theory-informed prompts help in the Davidson task. Gemini Flash consistently achieves the highest scores across tasks and corpora, while the Iser gap detection task remains substantially harder than binary truth-conditional judgments. Statistical tests confirm robust prompt effects and the failure of basic prompts. However, model behavior under incremental theory exposure is unstable and architecture-dependent.

BibTeX

@inproceedings{pichler2026a,
   author    = {Axel Pichler and Janis Pagel},
   booktitle = {{Proceedings of the 10th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2026)}},
   location  = {Rabat, Morocco},
   month     = {3},
   pages     = {280--294},
   title     = {{Evaluating Humanities Theory Alignment in Large Language Models: Incremental Prompting and Statistical Assessment}},
   url       = {https://aclanthology.org/2026.latechclfl-1.27},
   year      = {2026},
}

RIS

TY - CPAPER
TI - Evaluating Humanities Theory Alignment in Large Language Models: Incremental Prompting and Statistical Assessment
AU - Axel Pichler
AU - Janis Pagel
PY - 2026
J2 - Proceedings of the 10th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2026)
UR - https://aclanthology.org/2026.latechclfl-1.27
CY - Rabat, Morocco
SP - 280
EP - 294
ER -