NII Testbeds and Community for Information access Research
  • NTCIR Official site
  • Conference
  • Organizers
  • Data
  • Important Dates
Frequently Asked Qestions
  • What is 'Task Participant'?
    (What should 'Task Participant' do?)
    - Each Task Participant accomplishes research tasks using the data provided by NTCIR and submits the task results.
    - Every participant has to write and submit a paper for the Proceedings of the NTCIR Workshop Meeting and make a presentation.
  • What is the qualification for task participation?
    - We welcome every participant- member from a company, student, overseas or Japanese group, provided that each participant could submit the task results and the report on your research results.
  • What is a Workshop Meeting?
    - The Workshop Meeting is held every eighteen months and the research groups that participate in the tasks of the NTCIR Workshop present their experimental results and findings.
    The overview of the NTCIR Workshop and each task and the investigation conducted by the participating groups are presented at the Meeting.
  • What is 'Meeting Participant'?
    - 'Meeting Participant' attends the NTCIR Workshop Meeting. The Workshop Meetng is open to public. Each participating group in tasks is required to attend the meeting.
  • What is a Round Table Meeting?
    In the Round Table Meeting, task participants and anyone who will participate in the task discuss the Task Definition and the evaluation results.
  • Where do I find information about a NTCIR publication(the Proceedings)?
    - the Online Proceedings of the NTCIR Workshops are avilable at:
    - If you will obtain the proceedings in the printed version, please conact NTCIR Project Office at ntc-secretariat.
  • May I post my papers published in the NTCIR publications (the Proceedings)to an institutional repository?
    - Papers published in the NTCIR publications (Proceedings of the NTCIR Workshop Meeting and Proceedings of the EVIA Workshop) may be posted to an intsitutional repository with no embargo period and permission from NII is not required.
    -When you will post the papers, please be sure to pay attention to the following points.
    --Published source and the correct bibliographical information must be acknowledged.
    --Publisher's version/PDF* must be used.
    *'Publisher version/PDF' refers to the PDF with running titles, which used in the online proceedings on our website.
    -Reference: Contents of 'the NTCIR Workshop Copyright Form' signed by authors when the papers were published.
    --Copyright to the work is transferred to NII.
    --Authors may reproduce, or authorize others to reproduce their work, provided that the source and the NII copyright notice are included and the reproductions themselves are not offered for sale.
  • What is a Test Collection?
    -A 'test collection' is a data set used in system testing or experiments. In the NTCIR project the term "test collection" is used for any kind of data sets usable for system testing and experiments of IA technologies.

    An IR test collection consists of three components;
    (1) document collection
    (2) a set of topics: written statement of user's search request, and
    (3) relevance judgments: a list of relevant documents for each topic

    Evaluation of IA technologies is conducted as a relative comparison of the different approaches or strategies tested on the same test collection. The absolute values of scores of the experimental results of a system changes with the test collection used.

    An 'evaluation workshop' usually provides test collections and unified evaluation procedures for experiment results. Each participating group conducts research and experiments using the common data provided by the NTCIR organizer with various approaches.
    The importance of reusable large-scale standard test collections in IA research has been widely recognized and an evaluation workshop is now recognized as a new style of active research project that facilitates research by providing the data and a forum for research idea exchange and technology transfer.
  • How can I obtain the Test Collection?
    -To obtain the Test Collection, please submit (1) application form and (2) user agreement form (memorandum).
    For more information:
    - If you will use the Submission Archive, please contact the NTCIR Project Leader. For more information:
  • I am conducting research related to evaluation metrics of information retrieval systems. To verify the proposed method, can I use the submitted results(runs) in the past NTCIR Workshops?
    - The results(runs) submitted by task participants of the NTCIR Workshop are available for research purpose use after exchanging the contract between NII and the researchers as part of "Submission Archives", with the run IDs including the team IDs.
  • What is a submission archive?
    - Submission Archive consists of the results which were submitted by task participants of the NTCIR Workshop that NII organized(Raw Submission Data), Evaluation Data and Evaluation Tools.
    Submission Archive is the dataset to be used for research related to evaluation methods and metrics of information access systems. With regard to the purpose of use, please contact the NTCIR Project leader.
    For more information:
  • ow can I submit the user agreement forms(memorandums)?
    - Please send two original copies of the agreement forms by postal mail or courier. Since we need an actual signature, photocopied document without actual signature, faxed document and the scanned version in the PDF are not valid.
  • Where do I find the list of papers using NTCIR Test Collection?
    - A List of papers using NTCIR/on NTCIR is available at:
  • May I apply a patent on the research findings using the NTCIR Test Collection?-No problem. After laying opne the patent application, please submit a report as a publication of the research findings.
    How to submit the information of your publications:
    - NII is the academic group that the Article 30 of the Japanese Patent Law applies and you can apply a patent within six months from the date on which you make a presentation in the NTCIR Workshop Meeting, but we recommend that you have done the patent application procedure before the workshop meeting.
  • How can I subscribe the mailing lists?
    - Information on how to subscribe the NTCIR general mailing list is available at:
    http://research.nii.ac.jp/ntcir/ml-en.html Information on how to subscribe mailing lists for each task is available at
    the web pages of each task:
  • What is 'Organizer'?
    - An 'Orgnizer' is a member who is planning and organizing each task.
  • Who are current orgnizers?
    - The list of organizers is available at:
Last Modified:2009.09.10