Google DeepMind Pilots Double-Blind AI Evaluations: Neither Weights Nor Test Prompts Are Shared

Benchmark scores rest on an assumption: that the test questions did not leak in advance. What Google DeepMind announced on August 27, 2026 is a pilot that tries to back that assumption with cryptography rather than contracts and operating policy ¹. The company frames it as the world’s first double-blind evaluation of a proprietary, frontier-class AI model ¹.

The partners are the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons ¹. According to the technical report published alongside the post, the model actually evaluated was Gemini 2.5 Flash Lite, tested against a selection of the private AILuminate benchmark corpus using an NVIDIA H100 secure enclave on Google Cloud Platform, with OpenMined’s PySyft maintaining data and model privacy ².

The Student Who Saw the Answers

The technical report splits the trust problem in two. One half is deliberate: it cites a case of “benchmark hacking” in which one frontier lab tested 27 private model variants on Chatbot-Arena, then published the top scorer ².

The other half is accidental contamination, which the report calls a serious problem in its own right — and one that may not be solvable with trust, contracts and zero-logging policies alone. One study found strong evidence of benchmark data leakage in both pre- and post-training for roughly half of the 31 models it tested ². Another shows that test set contamination inflates measured benchmark performance, with the inflation growing alongside both the amount of contamination and model size ².

This is the same terrain covered by research on whether speech recognition models have memorized the test answers and by the effort to reproduce 2,226 ICML 2026 papers with agents. Just as NVIDIA’s perfect score on the public ARC-AGI-3 set came from rebuilding the harness rather than the model, scores move with the conditions of measurement. This pilot targets the step before all of that: whether the test questions were seen in advance.

Google says it already works with external partners — specialized research labs, civil society, national AI Safety and Security Institutes — to stress-test its models ¹. Zero-logging protocols and rigorous contractual safeguards have long kept external test prompts confidential, the company notes, but incorporating technical and cryptographic safeguards marks a step forward in secure evaluation ¹. The technical report goes further: evaluators cannot simply test proprietary models through the owner’s APIs, which expose benchmark prompts in plain text, and contractual arrangements with zero-logging systems could still fail to prevent inadvertent leaks into future training runs ².

Evaluating Without Either Side Seeing the Other’s Assets

Google frames the historical tradeoff plainly. Either evaluators handed over their testing prompts, risking the provider seeing the questions in advance, or the provider handed over model weights, risking its intellectual property ¹.

Double-blind evaluation splits that apart. Using Confidential Space within Google Cloud’s Confidential Computing portfolio, both the external evaluation data and the proprietary model can be cryptographically verified to remain private to their respective owners. The evaluator cannot see the Gemini model weights, and Google cannot see the evaluator’s test prompts ¹. The report describes the mechanism as secure enclaves that encrypt data in RAM at the hardware level ².

The setup is specific. Prompts were drawn from the reserve set of AILuminate (AIRR 1.4) — prompts that have never been processed by any model ². They covered critical hazard domains including chemical, biological, radiological, nuclear and explosive (CBRNE) hazards, cyberattacks, hate speech, self-harm and violent crime elicitation ². AVERI encrypted and decrypted the prompts and outputs, and AVERI staff evaluated the outputs ². The hardware was a Google Cloud A3 Confidential VM with Intel TDX host memory encryption, equipped with an NVIDIA H100 80GB Confidential GPU ².

Google says this cryptographic evidence helps prevent benchmark contamination and protects sensitive data ¹. As models grow more capable, the company argues, this matters especially for highly sensitive evaluations such as those used for cybersecurity or by government bodies, letting independent organizations test advanced models rigorously without compromising data sovereignty or security ¹.

The Limits the Report Names Itself

The part worth reading carefully is where the technical report states its own constraints.

Not all code could be inspected. Running Gemini 2.5 Flash Lite using only layers present in open source libraries was judged too significant an engineering challenge for this project, so proprietary method implementations could not be eliminated, and not all code could be inspected or allowlisted. AVERI was informed of this and accepted the overall setup with the enclave ².

Then there is the verification path. Reference values are published for the Confidential Space guest OS, its source is open, and its build pipeline is externally validated — but individual builds are not independently reproducible, because they take private signing keys as inputs. The team also relies on Google’s services to sign and verify the attestation report, which, as the report puts it, places Google in the verification path and increases the trust placed in it ².

The most practically telling point concerns where the bottleneck sits. The report states that the primary obstacle to secure double-blind evaluation is no longer hardware compute overhead, but the procedural overhead and human coordination required for legal agreements and code review ². The long-run goal is standardized cryptographic attestation pipelines that abstract away dependency hashes and keys with near-zero human overhead — an industry analogue, the report suggests, to the visual “HTTPS lock icon” on the web ².

Scaling comes next. Frontier models are growing beyond trillions of parameters, so evaluation will need distributed confidential clusters; expanding to many-node NVIDIA H100/B200 compute clusters with encrypted connections, alongside deferred execution frameworks that mask internal model architecture, is named as the next milestone ². Looking further out, the authors explicitly mark as speculation the idea that verification and attestation assurances could help securely evaluate advanced models on capabilities like cybersecurity: if a model tried to adjust its harness, weights or sandbox, the enclave hash would no longer match attestation and the instance would halt immediately ².

No Scores Were Published

What this announcement delivers is the mechanism and the fact of the run. The result itself — how Gemini 2.5 Flash Lite scored — appears in neither the blog post nor the technical report. What the pilot offers is not a conclusion about model safety but groundwork for reaching such conclusions.

For anyone choosing between models, this changes nothing immediately. The pilot covered Gemini 2.5 Flash Lite rather than a top-end model, and no general availability or pricing is indicated. What it does add is vocabulary: when comparing published benchmark numbers, there is now a way to ask whether those numbers came from a model that had not seen the answers. The lock-icon analogy the report reaches for is positioned as a goal that only starts to mean something once that question can be answered routinely.