What HUMANITAS evaluates

The lab addresses technologies that support the enrichment, analysis and interpretation of cultural heritage collections. The material includes, without being restricted to, historical newspapers, manuscripts, correspondence, literary works, visual artworks and multimodal collections.

These scenarios raise computational problems that standard benchmarks rarely capture: OCR and HTR errors, diachronic language variation, image classification over heterogeneous material, severe domain shift, and interpretative ambiguity in the ground truth itself.

Evaluation dimensions under discussion

The first edition treats these as open questions to be debated rather than as settled choices.

  • Standard NLP and IR metrics (precision, recall, F1, MAP, nDCG, CER and WER), and how suitable they actually are for DH tasks.
  • Information extraction performance, covering entity recognition, entity linking and relation extraction.
  • Robustness under OCR noise, multilingual transfer and diachronic variation.
  • Human-centred measures, including interpretability, trustworthiness, traceability and the communication of uncertainty.
  • Human-in-the-loop evaluation, including usability, expert annotation effort, and the degree to which a system supports an actual humanities workflow.

Why a new evaluation space is needed

Evaluation practice in DH is fragmented across communities, venues and projects. Initiatives such as CHiC, HIPE, HISEMOTIONS and PastReader have shown that demand exists, but their independence limits comparability between results, reuse of resources and long-term sustainability.

At European level, CLARIN and DARIAH share resources effectively, and neither provides a common framework for comparing those resources or the computational approaches applied to them.

HUMANITAS proposes a shared evaluation space where datasets, benchmarks, methodologies and user communities converge. It builds on the CLEF tradition in cultural heritage evaluation and extends it towards current DH concerns: information extraction, interpretation, human-in-the-loop workflows and the assessment of generative technologies.

The exchange runs in both directions. CLEF extends its evaluation culture into an active interdisciplinary field, and DH gains access to established benchmarking expertise.

What the first edition will produce

A shortlist of task scenarios and candidate datasets, an assessment of which evaluation methodologies transfer to humanities material and which do not, and a roadmap for the shared tasks of subsequent editions.

To complete: state explicitly whether the roadmap will be published as a written output (a lab overview paper in the CLEF proceedings, a white paper, a report circulated through CLARIN and DARIAH channels), since this affects what participants can cite afterwards.