Distribution Format Specification
- Question Format - Organizer will distribute topics(questions) in this format.
- Gold Standard Format - Organizer will distribute gold standard in this format
- Evaluation Result Format - TBA.
- Run ID Format - Run ID definition for Question Analysis, IR, End-to-End Question Answering, and Combination Run output submission
- Question Analysis Format - All the participants are recommended to submit question analysis results in this format
- IR Format - IR participants will submit their output (retrieval results) in this format
- End-to-End QA Format - End-to-End QA participants will submit their output (answers) in this format. Also use this format for the combination run. Basically the same as the GoldStandard Format.
- System Description - Short description about your system. TBA.
- Format Checker - TBA
- Center-Exam Scorer is available from the Download Web site for each Participating team. Sorry, currently the README is available in Japanese only. The English version will be provided later.
MiscQuestion Analysis Restrictions
You can submit up to 99 runs, but we may evaluate the first three runs only. We will select the runs to evaluate and analyze according to the "priority" specified in the Run ID.
We will accept up to 1,000 documents IDs, although we would evaluate the IR-effectiveness in terms of the effectiveness of the final QA results and we do not assess the relevance of the IR results themselves. IR results relevance may be able to assess later.
End-to-End and Combination QA Restrictions
You can submit up to 99 runs, but the first three run (PRIORITY = 01,02,03, ... ) should be evaluated.You can submit up to top 30 answers for each topic. However, due to resource constraints, we may not be able to evaluate all the answers in each topic.
See RunIDFormat for more details.
Distribution files are encoded in UTF-8. Please encode submission files in UTF-8 as well.