Scale AI and Singapore’s Infocomm Media Development Authority introduced SG-LegalClarify on November 12, 2026, to test legal reasoning under Singapore law and whether AI models recognise when they need more facts. The announcement reports evaluations of 19 models; Scale says a public benchmark subset is planned.
What SG-LegalClarify tests
The benchmark uses 1,100 English-language questions authored by lawyers across 10 legal domains. Of these, 760 omit facts needed to resolve the question. Each has a counterpart supplying those facts, bringing the total to 1,860 evaluation instances. A simulated user provides missing information when the model asks for it.
That pairing lets researchers compare answers to the same legal problem when information arrives upfront or must be elicited. The grading separates issue identification, rule recall, application and conclusion. The accompanying technical report describes 12,546 distinct lawyer-written rubric items and a three-model judging panel selected through comparison with human assessments.
This is the first research milestone from a collaboration formalised in April and announced by Scale in July 2026. The November development is the benchmark announcement and its findings, following the earlier commitment to develop a Singapore-law evaluation.
Search improved scores, but clarification remained difficult
Scale reports that the average mandatory-criterion score across the 19 models increased from 53.8% without web search to 63.0% with search. Recalling applicable legal rules remained a weakness: the web-enabled average for that dimension was 56.5%.
Without search, approximately 39% of scenarios with missing facts received no clarifying question, according to Scale. Even among web-enabled trials where models asked about every relevant missing fact, only 55.6% reached the correct conclusion. Information gathering and legal analysis therefore need separate interpretation.
The technical report defines several measurements with different denominators. The distinctions below matter when recording these findings in an AI evaluation card.
| Measurement | What it records | Interpretation for research teams |
|---|---|---|
| Mandatory score | The percentage of required rubric items passed in each trial, averaged across trials. | A criterion score does not state how many answers were completely correct. |
| Missing-fact coverage | The share of annotated hidden variables requested by the model. | Question frequency alone cannot establish whether the necessary facts were obtained. |
| Correct conclusion | Whether the response satisfies the conclusion rubric. | Obtaining facts and reaching a supported outcome remain distinct tests. |
What the findings mean for universities and research offices
For university researchers studying legal AI, the paired design offers a way to examine an assistant’s decision to ask questions alongside its eventual answer. For research offices assessing AI tools, our reading is that these results support closer attention to task coverage and the configuration of a model evaluation suite. Scores from this Singapore-law test do not establish performance on an institution’s own administrative questions.
The authors identify limitations that affect interpretation. Providers offered different search tools, so web-enabled results reflect both model capability and tool configuration. Errors in the simulated user or dialogue classifier can affect results, and the imposed routing between answering and gathering facts may differ from real interactions. The report explicitly warns against interpreting benchmark scores as evidence that a model is safe for unsupervised legal advice.
Singapore’s government Responsible AI Playbook provides related evaluation context: existing benchmarks may not fit a particular organisation’s risks, and automated judges need comparison with human labels. Together, these points make the announcement relevant as evaluation research, without establishing institutional approval for a legal assistant.
A public subset remains a planned release
Scale’s November announcement says it plans to release a public subset for research and independent analysis. The technical report also describes future release of evaluation code and model outputs for that split. Those materials would support scrutiny of computational reproducibility; the announcement does not state that the research package is already available.
The immediate contribution is a reported evaluation of jurisdiction-specific reasoning and selective clarification. Access to the planned materials will be a separate development for researchers seeking to reproduce or extend the work.
Sources
- Scale AI: Scale and Singapore’s IMDA Advance AI Evaluation with New Legal Reasoning Benchmark, November 12, 2026. Checked 2026-11-14.
- Gu and colleagues: SG-LegalClarify technical report, especially sections 2–4 and 6 and the ethics and reproducibility statements. Checked 2026-11-14.
- Scale AI: collaboration announcement with IMDA, July 7, 2026. Checked 2026-11-14.
- Singapore Responsible AI Playbook: Safety evals, benchmark fitness and evaluator validation. Checked 2026-11-14.
