MAST @ FIRE 2026 — Multilingual Agentic Search Track

About MAST

A cross-lingual agentic information retrieval challenge

Agents combined with search tools has demonstrated impressive capabilities on complex fact-seeking benchmarks such as BrowseComp [1]. However, existing search agent benchmarks are mostly in English — leaving the question unanswered: how well do agentic search systems perform when the user's queries are expressed in their potentially native and non-English languages?

MAST benchmark is focused on evaluation of multilingual agentic search systems. In our inaugral 2026 year, we are building on BrowseComp-Plus (ACL 2026) [2], a reproducible & verifiable extension of BrowseComp with 830 challenging English queries with a verified English corpus of ~100,000 web-sourced documents and human judgments. This year, we are translating queries across 15 typologically diverse languages potentially spoken by over 1.6 billion people.

This year's task is cross-lingual: queries are asked in one language, documents & answers are in English, and systems can reason, iteratively search, plan, and synthesize in either language. e.g., a search agent can ask a query in Chinese, use multilingual retriever to retrieve English documents for a given query in Chinese.

Next year, based on the feedback from this year's participants, we plan to extend MAST to use a multilingual corpus and human-written queries in 15+ languages.

📄 BrowseComp-Plus 📊 Leaderboard 💾 Data 💬 Slack

🌐

Multilingual Query

e.g., Chinese, Hindi, Swahili…

↓

🤖

Agentic Search System

Iterative retrieval + reasoning

↓

📚

English Document Corpus

~100K BrowseComp-Plus docs

↓

✅

English Answer

Short, verifiable, exact

Two Tracks in 2026

Choose the track that fits your research interest

Track 1

MAST Multilingual

Cross-lingual agentic search in 15 typologically diverse languages

Participating systems receive queries in one of 15 non-English languages spanning high, mid, and low-resource settings. The goal is to retrieve the relevant documents from BrowseComp-Plus corpus and produce a correct, concise answer in English.

🟢 High Resource Chinese · French · Russian · Spanish · German

🟡 Mid Resource Finnish · Arabic · Hindi · Bengali · Thai · Urdu

🔴 Low Resource Swahili · Tamil · Yoruba · Welsh · Māori

📝 ~100 queries per language
📚 ~100,000 English BrowseComp-Plus documents
📐 Evaluated on Recall (retrieval) + Answer Accuracy
🔁 Agentic runs (iterative) and retrieval-only both accepted

Track 2

MAST Indic

Deep-research agents for the Indian languages

A dedicated track for Indic languages. MAST Indic focuses on 10 Indic languages, morphologically rich languages, and spoken commonly in India.

🟡 Mid Resource (Indic) Hindi · Bengali · Tamil · Telugu · Kannada · Marathi · Gujarati

🔴 Low Resource (Indic) Punjabi · Odia · Malayalam

📝 ~100 queries per language across 10 Indic languages
📚 ~100,000 English BrowseComp-Plus corpus
🖋️ Covers 8 distinct scripts (Devanagari, Bengali, Tamil, Telugu, Kannada, Gujarati, Gurmukhi, Odia)
📐 Evaluated on Recall (retrieval) + Answer Accuracy
🤝 Collaboration with Indian NLP communities

Languages

Track 1 — MAST Multilingual: 15 typologically diverse languages

Language	Code	Family	Regions of Speakers	Script	Speakers (M)	Resource Tier
🟢 High Resource
Chinese	`zh`	Sino-Tibetan	China, Taiwan, Singapore, Malaysia	Han	1,351	High
French	`fr`	Indo-European	France, DR Congo, Canada, Belgium, Switzerland	Latin	332	High
Russian	`ru`	Indo-European	Russia, Belarus, Kazakhstan, Kyrgyzstan, Tajikistan	Cyrillic	199	High
Spanish	`es`	Indo-European	Spain, Mexico, Latin America	Latin	510	High
German	`de`	Indo-European	Germany, Austria, Switzerland, Liechtenstein	Latin	133	High
🟡 Mid Resource
Arabic	`ar`	Afro-Asiatic	Egypt, Algeria, Saudi Arabia, Iraq, Morocco	Arabic	375	Mid
Bengali	`bn`	Indo-European	Bangladesh, India	Bengali	280	Mid
Finnish	`fi`	Uralic	Finland, Sweden	Latin	6	Mid
Hindi	`hi`	Indo-European	India	Devanagari	584	Mid
Thai	`th`	Kra-Dai	Thailand	Thai	56	Mid
Urdu	`ur`	Indo-European	Pakistan, India	Arabic	313	Mid
🔴 Low Resource
Swahili	`sw`	Niger-Congo	Tanzania, Kenya, Uganda, DR Congo	Latin	194	Low
Tamil	`ta`	Dravidian	India, Sri Lanka, Singapore, Malaysia	Tamil	91	Low
Yoruba	`yo`	Niger-Congo	Nigeria, Benin, Togo	Latin	32	Low
Welsh	`cy`	Indo-European	Wales, Argentina	Latin	0.92	Low
Māori	`mi`	Austronesian	New Zealand	Latin	0.14	Low

Track 2 — MAST Indic: 10 Indian languages

Language	Code	Family	Country / Main Countries	Script	Speakers (M)	Resource Tier
🟡 Mid Resource (Indic)
Hindi	`hi`	Indo-European	India	Devanagari	584	Mid
Bengali	`bn`	Indo-European	India, Bangladesh	Bengali	280	Mid
Punjabi	`pa`	Indo-European	India, Pakistan	Gurmukhi	223	Mid
Telugu	`te`	Dravidian	India	Telugu	100	Mid
Marathi	`mr`	Indo-European	India	Devanagari	99	Mid
Tamil	`ta`	Dravidian	India, Sri Lanka, Singapore, Malaysia	Tamil	91	Mid
Gujarati	`gu`	Indo-European	India	Gujarati	65	Mid
Kannada	`kn`	Dravidian	India	Kannada	52	Mid
🔴 Low Resource (Indic)
Malayalam	`ml`	Dravidian	India	Malayalam	46	Low
Odia	`or`	Indo-European	India	Odia	45	Low

* MAST Indic covers 9 distinct scripts and both the Indo-European and Dravidian language families, making it one of the most script-diverse evaluation tracks for information retrieval.

Task & Evaluation

What systems must do, and how they are measured

🎯

Multilingual Retrieval & Reasoning

Given a query in a non-English language and the full BrowseComp-Plus corpus, participating systems must retrieve relevant evidence documents In English and produce a short and correct answer in English. Systems can be fully agentic involving open-source or closed-source systems and tools.

📐

Retrieval Evaluation

Every team must submit a list of retrieved document IDs per search turn. We will evaluate Recall against the human-verified relevance labels from BrowseComp-Plus, computed per language and aggregated per resource tier. We also evaluate search efficiency, i.e., the number of search turns required to reach the correct answer.

🧠

Answer Accuracy

Answer accuracy is evaluated using an open-source LLM judge (e.g., gpt-oss-120b) comparing the system's predicted English answer against the BrowseComp-Plus ground-truth answer.

📦

Submission Format

Runs are submitted as JSONL files containing reasoning steps, tool calls, retrieved document IDs per search round, and a final output answer. Maximum 3 runs per language per team.

View format spec →

Submission Format

JSONL output format shown below. Example submissions available on GitHub → BrowseComp-Plus.

output.jsonl

JSONL

{
  "query_id": 11,
  "language": "chinese",
  "retriever": "Qwen/Qwen3-Embedding-8B",
  "llm": "Alibaba-NLP/Tongyi-DeepResearch-30B-A3B",
  "tool_call_counts": { "search": 24 },
  "retrieved_docids": [
    ["81120", "10986", "11064", "83777", "9823"],
    ...
    ["81120", ...]  // kth search round
  ],
  "result": [
    {
      "type": "reasoning",
      "tool_name": null,
      "arguments": null,
      "output": "Let me search for information about..."
    },
    {
      "type": "tool_call",
      "tool_name": "search",
      "arguments": "query: \"As of December 2023\" coordinator research group founded in 2009",
      "output": "[{\"docid\": \"81120\", \"score\": 8.43, \"snippet\": \"...\"}]"
    },
    {
      "type": "output_text",
      "tool_name": null,
      "arguments": null,
      "output": "Explanation: I attempted to solve this ... Exact Answer: compassionate pugilist"
    }
  ]
}

No more than 3 output files per language per team. There is no restriction on the language used by the LLM during reasoning; however, direct translation of the original query to English as the only retrieval strategy is discouraged.

Working Notes

System description papers for FIRE 2026 proceedings

📄

Submission

Each participating team must submit one working note per track describing their system and approach. Working notes are submitted centrally via the FIRE 2026 submission system.

📐

Format

Working notes must follow the ACM two-column format (same template as SIGIR papers). Minimum 2 pages, maximum 4 pages excluding references. Use descriptive titles — do not include team or track names in the title.

- Overleaf Template

- Word Template

✍️

What to Include

System description: retriever, LLM, and agentic pipeline
Experimental setup and any training/fine-tuning details
Per-language and per-resource-tier results
Error analysis and qualitative observations
Limitations and future directions

📅

Deadlines

Aug 15, 2026 — Run submission deadline
Sep 15, 2026 — Working note draft due
Sep 30, 2026 — Camera-ready copies due
Oct–Dec 2026 — FIRE 2026 results session

⚠️ Author names should not include prefixes (Dr., Prof., etc.). At least one author per working note must be registered for FIRE 2026. Teams that submit runs but do not submit a working note may be excluded from the final leaderboard.

Schedule

MAST @ FIRE 2026 Tentative Timeline (subject to change)

June 10, 2026

Track website opens · Registration begins

June 15, 2026

Test data released for all languages

August 15, 2026

Run submission deadline

August 30, 2026

Declaration of results · Public leaderboard

September 15, 2026

Working note submission deadline

September 30, 2026

Camera-ready copies of working notes due

October 15, 2026

Overview paper · FIRE 2026 Conference

How to Participate

Three steps to join MAST

01

Register

Fill in the registration form with your team name, member names, affiliations, and the language(s) you plan to participate in. Teams are limited to 4 members.

Register →

02

Download Data

Access the test queries (released June 15, 2026) and the corpus through our GitHub repository and Hugging Face organization. Baseline retrieval runs will be provided.

Data (Jun 15) →

03

Submit Runs

Submit up to 3 JSONL output files per language before August 15, 2026. Submit a working note describing your system and approach (minimum 2 pages, ACM format).

Submission format →

Stay Connected

📬 Mailing List — for latest announcements and updates
💬 SIGIR 2026 Slack #fire-2026-mast — once registered for ACM SIGIR
🤗 huggingface.co/mast-benchmark — datasets & models
✉️ mast-organizers@googlegroups.com — email the organizers

Register for MAST 2026

Registration is free and open to academic groups, industry labs, and individual researchers. After registering, you will be added to the track mailing list and Slack workspace.