HUMANITAS is a lab at CLEF (Conference and Labs of the Evaluation Forum) devoted to evaluating and benchmarking computational methods that support research, access, analysis, enrichment, and interpretation across Digital Humanities (DH) and Cultural Heritage (CH). HUMANITAS is a backronym for Historical Understanding and Multimodal Analysis of Noisy, Interoperable Textual and Audiovisual Sources.

Where and when: CLEF 2027, Bucharest, Romania, 14–17 September 2027.

Scope and motivation · Programme · Organisers · Participate · Resources · Contact

In short

Advances in language technologies, multimodal models and generative AI open new possibilities for humanities research, and at the same time they make the absence of adequate evaluation frameworks more visible. Historical and cultural collections are noisy, multilingual and complex, and the questions asked of them often involve interpretation and uncertainty rather than a single correct answer.

CLEF has a long tradition in this area (CHiC for retrieval, HIPE for named entity processing), but DH benchmarking efforts remain scattered across venues (PastReader and HISEMOTIONS at IberLEF, among others). The European research infrastructures CLARIN and DARIAH have built solid ecosystems of resources, services and communities, yet neither offers a dedicated space for systematic evaluation across datasets, tools and workflows.

HUMANITAS is intended as a sustainable umbrella initiative within CLEF: a place where evaluation methodologies and datasets accumulate over time, and a benchmarking layer that complements CLARIN and DARIAH.

Goals

  • Define evaluation challenges for DH workflows.
  • Promote reusable, multilingual and historically grounded benchmark collections.
  • Consolidate and advance evaluation methodologies adapted to historical and noisy data, and to research tasks that involve interpretation and uncertainty.
  • Foster collaboration between DH scholars, technology researchers, CH institutions and research infrastructures.
  • Establish a sustainable framework for shared evaluation campaigns within CLEF.

First edition: a workshop, not yet a shared task

The first edition adopts a workshop format. Its purpose is community building, discussion of evaluation needs, assessment of available datasets and identification of priority challenges. Subsequent editions are expected to evolve towards campaign-style tracks with shared tasks and benchmark collections.

There is therefore no predefined shared task this year, and no system runs to submit. What the workshop produces is a roadmap: an agreed set of task scenarios, datasets and evaluation priorities that are suitable, impactful and sustainable for future CLEF campaigns.

Key dates

To complete: the whole table. CLEF labs normally align their calendar with the conference schedule (registration, working notes for CEUR-WS, camera-ready, conference). Check the official CLEF page for the corresponding edition and replace the placeholders.

Milestone Date
Call for contributions published TO COMPLETE
Submission deadline TO COMPLETE
Notification to authors TO COMPLETE
Camera-ready version (CEUR-WS) TO COMPLETE
Workshop TO COMPLETE

Who should come

Researchers in NLP, information retrieval and machine learning working with historical or noisy data; DH scholars in history, linguistics, visual arts and literary studies; professionals in libraries, archives and museums; developers and data scientists working in diachronic and interpretative settings.

Organised in dialogue with

CLARIN, DARIAH, the HIPE and CHiC evaluation communities, and the projects and infrastructures represented in the organising team.

To complete: decide whether logos of the participating institutions and infrastructures should appear here, and obtain permission to use them. Add the image files to an /assets folder and link them from this page.