NTCIR

NII Testbeds and Community for Information access Research
  • NTCIR Official site
  • Conference
  • Organizers
  • Data
  • Important Dates
 

Run Formats (NTCIR-11 Temporalia)


Please read our important notification before running formal run data with your system.


To submit the outputs of your system (called "runs"), you need to follow the run format defined as below. The runs formatted in a wrong way will not be assessed by the organizers.


Temporal Query Intent Classification (TQIC) subtask:

id     class     group_id     run_id

001     future     roi_y     roi_y_system1

002     past     roi_y     roi_y_system1

...

Each line should contain tab separated result for a different query. "id" is query identifier, while "class" represents the temporal class of query's intent and should be one of: past, future, recent, atemporal. "group_id" is an identifier of the group submitting the run and "run_id" is the run number for the group. For example, if you have two systems with different algorithms or parameters, please use a different run ID for different system settings.


Each group can submit up to three runs (i.e., Run IDs) for the TQIC subtask.



Temporal Information Retrieval (TIR) subtask:

id     rank     doc_id     group_id     run_id

001a     1     DOC1     roi_y     roi_y_system2

001p     2     DOC2     roi_y     roi_y_system2

...

Each line should contain tab separated result containing relevant document to a given subtopic. You can return up to 100 documents for each subtopic.


"id" is the query id, which is a concatenation of the query id and "p", "a", "r" or "f" that indicate the past, atemporal, recent and future subtopics, respectively. "rank" is the rank of the document for that query, "group_id" is an identifier of the group submitting the run and "run_id" is the run number for the group.  For example, if you have two systems with different algorithms or parameters, please use a different run ID for different system settings. Documents are identified by doc_id.


Each group can submit up to three runs (i.e., Run IDs) for the TIR subtask.



File name and format

Please use one file per subtask, and name your files with the subtask name and group id (e.g., tqic_roi_y and tir_roi_y). You can append multiple runs in a single file. Please use one file per run ID for formal runs. Regardless of the number of subtasks you're submitting, please zip all the files, and name it as "group_id.zip" before uploading to the submission system.