Retrieval over a Biomedical Dataset Database: Tokenization, Terminology Expansion, and Evaluation

1. The problem

What is searched. A few dozen to a few thousand dataset cards, each with a name, aliases, a bilingual description, tags, a license verdict and evidence. The corpus is small, so recall problems come from vocabulary mismatch, not from volume.

Query forms seen in practice. Chinese phrases without spaces ("医患对话"), English phrases, mixed names ("TCM 问答", "cMedQA 问答"), hyphenated and cased variants of dataset names, misspellings of names, synonyms across languages ("Traditional Chinese Medicine benchmark" versus "中医评测集"), and queries for things that are not in the database.

What went wrong. Three problems were observed in the first run of the agent benchmark and in an earlier real-client test (internal records). First, a rule that every word of a multi-word query must match returns nothing for natural queries: of the 95 searches in the 30 tasks, 13 returned zero results, five of them multi-word queries; most of the rest asked for something the database does not contain. Second, an agent looking for the national drug-reimbursement list dataset by its short name needed three keyword attempts, because the short name was not matched against the identifier and alias. Third, agents did not know when to stop: a first search that had returned the full set was followed by further searches that only confirmed it. The last problem concerns the response format and is covered in a companion note (research/notes/N1-agent-database-interface); the first two are retrieval problems.

2. Why semantic similarity is not enough

A natural idea is to rank by semantic closeness of the query to a card's description. For this database that is risky for two reasons. First, the objects are named things: a dataset has a name, aliases and an identifier, and each is attached to one license verdict. A query for one named dataset that returns a similar-sounding neighbour could lead an agent to quote the wrong license, which is the worst error this database exists to prevent. Exact and alias matches must therefore outrank approximate ones, and an approximate match must be visible as such. Second, close terms such as a benchmark and a training set, or a corpus and a knowledge graph, matter for the task at hand, and a general-purpose similarity measure may blur them (not tested here). Semantic methods may still help as a late-stage fallback; no vector retrieval is part of the current method.

3. Method

Tokenization. One tokenizer is used for both indexing and querying, so the two cannot disagree. Chinese text is split into overlapping character pairs and matched as phrases, which needs no dictionary and copes with names and new terms; the cost is that single-character queries match widely and the index is somewhat larger. Latin text is lower-cased, and hyphen, space, case and full-width variants of names are normalised so that variants of one name reach the same card. A query phrase that is not found verbatim may still match when all its character pairs occur in one card. A small edit-distance tolerance handles misspelled names.

Ranking. Matches in the name, aliases and identifier outrank matches in the description or tags, and an exact name, identifier or alias match receives a further boost. A small prior favours recently verified cards with ample evidence and can be switched off per request. A result can be returned with an explanation of its score.

Multi-word queries. A multi-word query ranks cards by the number of words matched. Cards matching all words come first; cards matching at least half of the words follow and are marked as partial matches. A strict all-words mode remains available per request.

Synonym expansion, kept narrow. A maintained file of synonym groups (29 groups) expands a query term to the other terms of its group, and synonym hits rank below exact hits. The rules for the file: only terms that mean the same thing in this domain; no broader terms ("medical"); no ambiguous abbreviations (for example KG, GB, CT); every group states its source; specialty and ICD-11 chapter names are added from the vocabulary automatically; expansion can be turned off per request. The aim of the narrow scope is that expansion should not manufacture matches that make an agent believe a dataset fits when it does not.

Empty and complete results. If nothing matches, the response says so explicitly and says how many records each filter alone would have excluded, so that the agent can stop. When a search returns the full set, the response says that the list is complete. For checking a list of known datasets, an anonymous request can pass the identifiers directly and receives the unknown ones reported back, instead of one search per item.

4. How quality is measured

A regression set of 85 queries (an internal test file) was written from dataset names, aliases and cards, not from the engines' output, and covers 13 kinds of query (Table 1); 26 of them are real searches taken from the benchmark call logs. Each query lists the datasets a reasonable user would want, or is marked empty when the database has nothing and the correct answer is no result (8 queries; 5 of them form the kind "empty", the other 3 sit in other kinds). The evaluator reports, per kind and overall: the share of queries with a relevant dataset in the first five results (for empty queries, no result returned), mean reciprocal rank (MRR) and precision at five (P@5). Three engines are compared on the same cards: an embedded full-text engine, a pure-code BM25 engine and the earlier substring filter. The evaluation uses the 43 curated cards of the first-release catalogue; cards added later by the automated pipeline are excluded by a fixed rule. An accepted baseline is kept, and the test suite fails if a later change to the tokenizer, synonyms, weights or priors lowers a number.

Table 1. Query kinds in the regression set

Table 2. Overall results on the 43 curated cards (85 queries)

Table 3. Top-5 hit rate / MRR by query kind

5. Results

All figures are produced by scripts/retrieval-stats.mjs, which runs the regression set and compares the results with the accepted baseline (they match). Tables 2 and 3 give the results.

On the 43 curated cards, both new engines return a relevant dataset in the first five results for every query (top-5 hit rate 1.0), with MRR 0.986 (full-text) and 0.994 (BM25) and P@5 of 0.84 and 0.845; the earlier substring filter reaches 0.706 on top-5 hit rate and MRR and 0.662 on P@5. The gap is largest for misspelled names (5 queries: 1.0 against 0.0), disease-category queries (2: 1.0 against 0.0) and hyphen variants of names (9: 1.0 against 0.667). Only the synonym kind (MRR 0.833 and 0.875 on 4 queries) and, for the full-text engine, the benchmark-derived kind (0.981) fall below 1.0 in MRR; synonym hits rank below exact hits by design. All 8 queries whose correct answer is "no result" (marked empty in the set; 5 of them form the kind "empty") return no result on all three engines. In the first benchmark run, before these changes, 13 of 95 searches returned no result; a repeat of the benchmark on the revised interface will be reported with the revised benchmark report, because single runs of an agent are noisy and not conclusive.

6. Limitations

• The set favours the engines. The queries were written by the team that built them, from dataset names and cards, and are optimistic; a top-5 hit rate of 1.0 means the known cases pass, not that unseen queries will. The set serves as a regression guard.
• Saturation. With top-5 hit rate at 1.0, only MRR and P@5 still separate the new engines; they cannot be told apart reliably on 85 queries.
• Small set, few queries per kind. Kinds have 1 to 26 queries.
• Curated cards only. The evaluation uses the 43 curated cards; with more cards in the catalogue, partial matches become more frequent and results will differ.
• Narrow synonym file. 29 groups; no vector retrieval; the file has not been reviewed by a second person.
• Order of words. Ranking by number of matched words cannot tell which word of the query matters more.
• Benchmark-derived queries come from one low-cost model's searches.

7. Reproducibility

scripts/retrieval-stats.mjs accompanies the note. It needs the retrieval code and the regression set, which are internal; they are available from Mdexa on request and are not part of the public database repository. Run without arguments it only checks its output against the committed figures.

Engineering note, not peer reviewed. Mdexa Engineering, 2026-11-13. Based on internal test files and the agent benchmark records, which are not public.

How to cite

Mdexa Engineering. Retrieval over a Biomedical Dataset Database: Tokenization, Terminology Expansion, and Evaluation. Engineering note, not peer reviewed. 2026-11-13. Data version: regression set of 85 queries evaluated on the 43 curated first-release cards; first benchmark run of 2026-11-13.