NTCIR

NII Testbeds and Community for Information access Research
  • NTCIR Official site
  • Conference
  • Organizers
  • Data
  • Important Dates
 

NTCIR-11 QA-Lab

updated on 2014-09-24


RunID Definition for QA-Lab runs (End-to-End and Module output)



Eash run has to be associated with a RunID which is an identity for each run and use the RunID as the filename of the run results. Based on NTCIR CLIR and ACLIA format as a base, the RunID is defined as follows;

 [Topics' (Questions') File Name without the Extension (.xml)]_[Team ID]_[Language]_[RunType]_[Priority].[FileType]

Two character language codes are as follows;
  • EN (English)
  • JA (Japanese)
Two character RunType codes are as follows;
  • QA (Question analysis module output) (XMI and/or XML)
  • RS (Information Retrieval result) (XMI and/or XML)
  • IX (Information Extration from retrieved documents) (XMI only)
  • FA (Final Answer of the End-to-End QA and Combination Run output) (XML only)
  • SD (System Description Form) (Text file)

Priority Parameter;

The "Priority" is two digits used to represent the priority of the run, taking 01 as the highest. It will be used as a parameter for pooling and priority to be evaluated and analysed the results. Upto three runs will be evaluated and analyzed. But the number of the runs included in the evaluation may vary according to the total number of submissions.


File Type:

Please indicate the file type using "xmi" or "xml".

For System Description Form, please use "txt"


For Combination runs, please list all the Team ID's used in the run.

 [Topics' (Questions') File Name without the Extension (.xml)]_[IR Team ID]_[Question Answering Team ID]_[Language]_[RunType]_[Priority].[FileType]

For example, suppose TEAM1 submitted an Information Retrieval result with a RunID:
 Center-2009--Main-WorldHistoryB_TEAM1_EN_RS_01.xml

If Question Answering participant TEAM2 uses this result as an input to the system, and TEAM2 submits the combination run with a following RunID;
 Center-2009--Main-WorldHistoryB_TEAM1_TEAM2_FA_01.xml