The evidence passes

S1 Which instrument reaches the reader depends on two things, and neither is the document

regulator

Not the sensitivity of what is in the file. Who is holding it, and where. See the table above.

OPC, Questions and Answers regarding the application of PIPEDA, Alberta and British Columbia's Personal Information Protection Acts, and Summary of privacy laws in Canada.

S2 Responsibility does not transfer with the data

regulator

"Organizations are responsible for personal information in their possession or custody, including information that has been transferred to a third party for processing."

The guidance recommends that an organisation, before engaging a provider, "confirm in a contract the provider's understanding of their roles and responsibilities and of their obligations under PIPEDA," and assess data flows and locations, security controls, subcontractors, retention and deletion, and ongoing monitoring.

And it speaks to AI tools directly: it recommends that organisations understand what the technology actually does, use "extra scrutiny where a third-party service provider claims that the information involved is anonymized", confirm what the training data is, and assess lock-in.

OPC, Guidance on assessing third-party service providers, 10 September 2026.

This is the single most useful source in the pass and it is three days old.

S2a PHIPA binds the recipient, and it binds them to the original purpose

statute

The operative limit on an Ontario practitioner is not a general privacy duty. It is purpose: a non-custodian who receives personal health information from a custodian may use it only for the purpose the custodian was authorised to disclose it for, or to carry out a statutory or legal duty — and may not use more of it than is reasonably necessary.

A file disclosed so that an employer could assess accommodation was not disclosed so that a summarising tool could be tried out on it. The question stops being is this personal enough to matter and becomes is this the purpose it came to me for.

Ontario PHIPA, s. 49(1) and (2), from e-Laws.

S2b Ontario's own de-identification standard, and it is not "take the name off"

statute

"'de-identify', in relation to the personal health information of an individual, means to remove any information that identifies the individual or for which it is reasonably foreseeable in the circumstances that it could be utilized, either alone or with other information, to identify the individual."

Either alone or with other information. A rib fracture on a named date, at a named employer, with a named union and a return-to-work date is not de-identified because the name came off — the rest of it identifies the person to anyone who has the file.

Ontario PHIPA, s. 2 definition of "de-identify".

S3 Sending information outside Canada is not prohibited — accountability simply stays

regulator · located, not read at source

PIPEDA does not bar transferring personal information abroad for processing. Canada took an organisation-to-organisation approach rather than the EU's state-to-state adequacy model: there is no list of approved countries. Instead the transferring organisation must use contractual or other means to provide a comparable level of protection while the third party processes it.

This is consistent across the OPC pages found but the underlying records were not independently opened in this pass.

S4 Quebec requires destruction or anonymisation once the purpose is achieved

statute

"Where the purposes for which personal information was collected or used are achieved, the person carrying on an enterprise must destroy the information, or anonymize it to use it for serious and legitimate purposes, subject to any preservation period provided for by an Act."

Anonymisation must follow "generally accepted best practices and … the criteria and terms determined by regulation." That regulation exists — A-2.1, r. 0.1 — and was not read in this pass.

S5 Quebec's automated-decision provision, verbatim — independently confirmed

statute

"Any person carrying on an enterprise who uses personal information to render a decision based exclusively on an automated processing of such information must inform the person concerned accordingly not later than at the time it informs the person of the decision."

On request, the person must also be told the information used, "the reasons and the principal factors and parameters that led to the decision," and their right to have that information corrected — and "must be given the opportunity to submit observations to a member of the personnel of the enterprise who is in a position to review the decision."

C1 "The law will tell me whether I can put this in"

For a large share of her readers, on employee information, there is no privacy statute to consult.

C2 "I took the name off, so it is de-identified"

Quebec's standard is irreversible, direct or indirect, at all times reasonably foreseeable (S4). And the new OPC guidance tells organisations to use "extra scrutiny where a third-party service provider claims that the information involved is anonymized" (S2). A claim file describing one person's injury, employer, union and treating physician is not anonymous because the name is gone.

C3 "Once I have sent it, it is the vendor's problem"

Directly contradicted by S2. Possession or custody follows the organisation.

C4 "My employer has a policy, so I am covered"

inference from S2, not a quoted finding

The obligations in S2 sit on the organisation and are discharged through contracts, assessments and monitoring of a provider the organisation chose. A practitioner pasting a file into a personal account is outside whatever arrangement exists — not covered by it. [inference from S2, not a quoted finding]

N1 Whether any particular paste is lawful

no instrument

N2 How far s. 49's "purpose" stretches in practice

no instrument

PHIPA s. 49 is read at source and quoted above, so the instrument is established. What is not established is how broadly "the purpose for which the custodian was authorized to disclose" is read in practice — whether, for instance, summarising a file to assess the same accommodation is the same purpose or a new one.

N3 Alberta and BC PIPA employee-information provisions

located, not read at source

Consistent across the search pass; the statutes were not opened.

N4 The Quebec anonymisation regulation's criteria

not checked

A-2.1, r. 0.1 exists and s. 23 points at it. Its contents were not read.

N5 Nothing in any Canadian privacy statute addresses AI tools by name

no instrument

The instruments are technology-neutral. The only AI-specific material found is regulator guidance — the 10 September 2026 OPC document — not statute.

S1 Fluency and completeness trade against each other, and reviewers do not correct for it

peer-reviewed

Strongest single citation in this file.

Cross-sectional study, 100 inpatient hospital-medicine encounters, UCSF, blinded duplicate review by 22 attending physicians. LLM- vs physician-generated discharge summary narratives:

Reviewer preference did not differ (χ² = 5.2, P = .27). Harm potential per individual error did not differ (1.35 vs 1.34, P = .99); overall narrative harm score was higher for the LLM but low in absolute terms for both (0.84 vs 0.36 on a 0–7 scale, P < .001).

What you can say: the machine writes tighter and reads better, and it is measurably less complete — and a room of attending physicians, reading blind, rated it just as good and picked it just as often. The quality that makes it persuasive is the same quality that makes it lossy, and expertise did not correct for it.

Several other studies carry near-identical titles about LLM discharge summaries.

S2 Omission and fabrication land in different parts of the document

peer-reviewed

The most operationally useful finding in this file.

100 randomly sampled adult ED encounters, UCSF, 2012–2023. GPT-4 and GPT-3.5-turbo, two independent physician reviewers, three error criteria.

• GPT-4: 33% of summaries entirely error-free. Inaccuracies 10%. Hallucinations 42%. Omissions 47%.

• GPT-3.5-turbo: 10% entirely error-free.

• Mean harm score across errors 0.57 / 7; three errors scored ≥ 4.

• Inaccuracies and hallucinations concentrated in the Plan section. Omissions concentrated in Physical Examination findings and History of Presenting Complaint.

Why this matters more than the headline percentages: the two failure modes are not spread evenly. It invents in the recommendation. It loses in the findings and the history.

The 42% / 47% figures circulate widely from the 2024 medRxiv preprint (PMID 38633805), which is not peer-reviewed. The peer-reviewed article above carries the same numbers. Industry white papers routinely cite the preprint.

S3 AI drops the scope conditions — and gets worse, not better, when told to be careful

peer-reviewed

10 LLMs (ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3 70B, Claude 3.7 Sonnet and earlier versions), 4,900 summaries compared against their source texts.

• Overgeneralisation in 26–73% of cases for DeepSeek, ChatGPT-4o and LLaMA 3.3 70B.

• Against human-authored summaries: odds ratio 4.85, 95% CI [3.06, 7.70], P < .001 — nearly five times more likely to generalise beyond what the source supports.

• Newer models performed worse than earlier ones.

• This held even when the models were explicitly prompted for accuracy.

What you can say: this is scope loss, measured. "Unfit to work as at 2 October, reassess in *fourteen days*" becomes "unfit to work." The qualifier that bounds the finding is the part that goes. And the two instincts a careful practitioner would reach for — use a newer model, tell it to be accurate — are the two the evidence says do not work.

Caveat: the source texts were scientific abstracts and full-length medical articles, not long evidence-dense files. The mechanism transfers; the rates were not measured on your material.

S4 Attribution collapses faster than content

peer-reviewed, non-PubMed

SummHay: ten ~93,000-token "haystacks," ~62 planted insights each, each insight present in ≥5 documents. Two scores — did the model recover the insight, and did it attribute it to the right document?

Content survives about half the time. Provenance survives about a fifth. For a practitioner reading a file where which doctor said it, on what date, on which form is the whole question, this is the loss that matters most and the one nobody warns about.

Caveat: synthetic haystacks, multi-document. Not single-document source-voice separation.

S5 Position determines what survives — in the source and in the summary

peer-reviewed, non-PubMed · preprint

• Middle of the source is worst. Summarisation-specific study across 6 models and 10 datasets, with a purpose-built set (MiddleSum) where no salient content sits at the start: U-shaped utilisation. Ravaut M et al., ACL 2024, 2764–2781. aclanthology.org/2024.acl-long.153

• End of the summary is worst. Unfaithfulness concentrates in the later portions of generated output; human-written reference summaries dip around the middle but recover. [preprint] — arXiv:2505.15291. Corroborated independently by Belem et al., arXiv:2410.13961.

• Foundational: Liu N et al., Lost in the Middle, TACL 2024;12:157–173 — a QA finding, which is why the summarisation-specific study above matters.

What you can say: the bottom of a long AI summary is less trustworthy than the top, and the middle of a long file is where things disappear. Both are actionable at the desk.

S6 Verification complexity is exactly the condition that produces automation bias

peer-reviewed

Systematic review, 890 papers screened → 40 included. Against the prevailing human-factors view, automation bias was found in single tasks, typically diagnosis rather than monitoring, and with high verification complexity. Associated with cognitive load, not uniquely with multitasking.

Checking an AI summary against a long claim file is a single, high-verification-complexity, judgement-forming task. That is the exact configuration the review identifies as most bias-prone. The practitioner is not careless; the task is shaped to defeat care.

S7 A biased model costs about four times what a good model gains — and explanations do not rescue it

peer-reviewed

Randomised clinical vignette study, 457 hospitalists, NPs and PAs across 13 US states.

• Baseline diagnostic accuracy 73.0% (95% CI 68.3–77.8)

• Standard AI, no explanations: +2.9 pp (95% CI 0.5–5.2)

• Standard AI, with explanations: +4.4 pp (95% CI 2.0–6.9)

• Systematically biased AI: −11.3 pp (95% CI 7.2–15.5)

• Biased AI with explanations: −9.1 pp — a non-significant 2.3 pp recovery (95% CI −2.7 to 7.2)

What you can say: the asymmetry is roughly four to one against you, and the standard regulatory answer — make the model explain itself — was tested and did not work.

S8 The evidence was already unreliable before any AI touched it

peer-reviewed

• Median inter-rater reliability among disability assessors in the insurance setting: 0.45 (range ICC 0.86 to κ −0.10). Poor in 6/16 studies (37%), excellent in only 2/16 (13%). In the research setting — same task, no normative or legal dimension — median 0.76, with 5/7 (71%) excellent. Reliability was higher when a standardised instrument guided the evaluation (P = .006). 23 studies, 12 countries. Barth J, de Boer WEL, Busse JW (McMaster) et al. BMJ 2017;356:j14. PMID 28122727 · DOI

• Reading the file is itself unreliable — the closest analogue that exists. 12 insurance physicians assessed work limitations for 12 patients from written reports only. Mean agreement 76% (range 64–88%), but agreement on the number of hours a patient could function daily was low. Only half the physicians believed a reliable assessment from written information was possible; 25% said it was not. Spanjer J et al. Disability and Rehabilitation 2008;30(24):1885–1890. PMID 19037781 · DOI

• FCE barely predicts return to work — and it is Canadian data. Historical cohort, Alberta workers' compensation claimants, Isernhagen Work Systems protocol. Only 4% passed all tasks, yet most had benefits suspended and claims closed within the year. Better performance related to faster recovery but explained roughly 10% of the variation. One task — floor-to-waist lift — was as predictive as the entire protocol. Gross DP, Battié MC, Cassidy JD. Spine 2004;29(8):914–919. PMID 15082996 · DOI

The argument is not "AI is bad at reading this evidence." It is: the evidence is contested to begin with, two qualified humans reading the same file disagree about it, and a summarisation layer places a fluent, confident, unattributed account on top of material that was never that certain — in a task shaped to defeat verification. That argument survives a well-read opponent. "AI makes mistakes" does not.

S9 There is a validated instrument

peer-reviewed

PDSQI-9 — Provider Documentation Summarization Quality Instrument, purpose-built for LLM-generated clinical summaries. 7 physician raters, 779 summaries, 8,329 questions. Cronbach's α 0.879 (95% CI 0.867–0.891); ICC 0.867. Four-factor structure explaining 58% of variance: organization, clarity, accuracy, utility. Discriminated high- from low-quality summaries (P < .001).

C1 "The AI is worse at this than a person" — no

peer-reviewed

• JAMA Internal Medicine (S1): comparable overall quality, equally preferred by 22 blinded attending physicians.

• Preconsult summaries in breast surgical oncology, 200 consultations: the LLM beat clinical staff on nodal status (95% vs 72%, P < .001) and invasive component on biopsy (94% vs 30%, P < .001). Staff beat the LLM on receptor status (96% vs 86%, P = .002). Patient-level fabrication 8% for the LLM vs 3.0% for staff; clinically significant 3% vs 0.5%. Authors: errors were "largely confined to identifiable high-risk variables that are amenable to targeted verification." Moo TA et al., JCO Clinical Cancer Informatics 2026;10(3):e2600160. PMID 42691437 · DOI (Published 3 September 2026 — one day before this pass.)

The defensible claim is about error type and verifiability, not about the machine being worse.

C2 "Three places it reliably loses the thing that mattered" — the evidence does not give you three

peer-reviewed · no evidence

What actually exists:

C3 "Prompt it carefully and it's fine" — explicitly refuted

peer-reviewed

• Peters & Chin-Yee: explicit accuracy prompting did not prevent overgeneralisation, and newer models were worse.

• CREOLA. 18 experimental configurations, 12,999 clinician-annotated sentences, GPT-4, primary-care consultation transcripts. Hallucination rate 1.47%; omission rate 3.45%. Across prompt configurations, the best produced 2 major hallucinations and 0 major omissions; another produced 1 major hallucination and 10 major omissions. Prompt engineering trades the two failure modes against each other rather than removing them. Hallucination subtypes: fabrication 43%, negation 30%, contextual mixing 17%, causality 10%. Asgari E et al. npj Digital Medicine 2025;8(1):274. PMID 40360677 · DOI

All seven authors were employees of Tortus AI when the paper was written (one part time), a commercial clinical-scribe vendor, and the paper's own conclusion is that refining prompts "successfully reduced major errors below previously reported human note-taking rates" — a vendor claim in a peer-reviewed wrapper. That does not make the error counts wrong; peer review at npj Digital Medicine is real. It does mean the most-cited omission figure in this field comes from a company selling the tool, and the direction of any bias would be toward understating harm.

Thirty per cent of GPT-4's hallucinations in a clinical-documentation task on mock primary-care consultations were explicit contradictions of a clinical fact — the "no evidence of" / "unfit to work" class.

C4 Two things circulating in this field that are not what practitioners think

preprint · grey

• The 42% / 47% ED figures are usually cited from a preprint.

• AIDA is not law. The Artificial Intelligence and Data Act died with Bill C-27 on prorogation in January 2025. Canada has no general federal AI statute. Practitioners routinely assume otherwise.

C5 Statistics Canada figure

government

Labour Force Survey supplement, March 2026, published in The Daily 30 July 2026: 35.9% of workers aged 15–69 used generative AI at work in the previous 12 months. Finance, insurance and real estate: 59.2%. Professional/scientific/technical: 65.6%. Legislative and senior management occupations: 75.1%. Daily use: 31.4%.

(Both figures read directly from the Statistics Canada page on 4 September 2026, not from a secondary source.)

N1 Nothing exists on AI summarisation of functional-abilities evidence

no evidence

Probed from six directions: fitness-for-work, workers' compensation NLP, disability claim adjudication, RTW prediction, insurance claims ML, occupational health LLM. Zero peer-reviewed studies on LLMs summarising Attending Physician Statements, Functional Abilities Forms, or FCE reports. Zero on AI in disability-insurance or workers' compensation adjudication. Zero Canadian. No research on the Ontario WSIB Functional Abilities Form (2647A) or on APS quality.

The one direct hit — Vella F et al., Frontiers in Public Health 2026;14:1815316 · DOI — reports 100% concordance between an LLM and a designated occupational physician, supported by collegial discussion with two additional occupational physicians, at one firm. A reported 100% concordance is a red flag, not a validation. Do not lean on it.

This is a finding, not a search failure.

N2 Chronology and sequence loss — unmeasured

no evidence

No study measures chronology collapse, date loss, or event merging as an outcome of summarising a long document. The only taxonomy that names it (FRANK, NAACL 2021) reports 13% "circumstance errors" covering wrong time or place, on pre-LLM models, and its "discourse link error" category — which explicitly includes incorrect temporal ordering — accounts for 3% of errors across all models. SummHay's authors list temporal relationships among the things their benchmark deliberately excludes.

For a document where when something was said changes its meaning, there is nothing to cite.

N3 Dropped numbers and thresholds — never isolated

no evidence

Every taxonomy folds numeric errors into "entity errors" (FRANK 36%, SummEdits 78% entity modification) without separating them. The one dedicated paper — Zhao et al., "Reducing Quantity Hallucinations," Findings of EMNLP 2020 — is pre-LLM. No modern study reports a rate for dropped or altered weight limits, durations or thresholds.

Counter-signal worth knowing: one small 2025 study on a legal/news corpus found modern models "did not invent entities or misstate numbers" — the failures were interpretive. n=40. Treat as hypothesis.

N4 Contradiction-smoothing — untested, and it is Exhibit B's exact question

no evidence

No study places two genuinely conflicting statements in a source document and measures whether the summary preserves the conflict, flags it, or resolves it into false agreement.

The nearest evidence:

• ContraDoc (NAACL 2024): GPT-4 detects document-internal self-contradiction at 53.8% — chance, on a balanced set — and on documents under 2,200 tokens. Human annotators did worse: 32% false-negative rate.

• CREOLA: 17% of GPT-4's hallucinations were "contextual" — information from separate contexts inappropriately mixed.

• Dahl et al., Journal of Legal Analysis 2024;16(1):64–93 · DOI: models accept false premises supplied by the user at high rates (GPT-4: 69% on a false-dissent premise). This is deference to a premise, not smoothing of an internal conflict. Do not conflate them.

N5 Whether human reviewers catch omissions — no human-subjects study

no evidence · preprint

No study compares human catch-rates for omissions versus fabrications in AI summaries. The strongest available signal is an LLM-judge asymmetry — commission detection 0.79–0.94, omission detection 0.50–0.63 against a chance floor of 0.50 (arXiv:2608.31016, [preprint], corresponding author's contact at composo.ai). Plus the structural argument: absence is not a checkable proposition. You can verify a sentence that is present. You cannot verify a sentence that is not there without re-reading the source, which is the step the summary was meant to save.

That is a sound basis for a design decision and a weak basis for a statistic.

S1 Human-plus-AI is worse than the better component on decision tasks

peer-reviewed

[peer-reviewed] The strongest single citation in this file.

Systematic review and meta-analysis, 106 experiments, 370 effect sizes, 74 papers, Jan 2020 – Jun 2023.

combining a person and a model is not additive, and on decision tasks it is subtractive. The combination helps when the person was already the better decider — and hurts when the model was. Anyone deploying AI on a decision has already asserted the model is good. That is the losing quadrant.

Vaccaro M, Almaatouq A, Malone T. When combinations of humans and AI are useful: a systematic review and meta-analysis. Nature Human Behaviour 2024;8(12):2293–2303. PMID 39468277 · DOI · free full text at PMC11659167.

Verified against the PubMed record on 4 September 2026 — the headline figure is in the abstract; the rest are from the full text.

S2 Oversight produces differently-distributed error, not better outcomes

peer-reviewed · preprint

[peer-reviewed] + [preprint] Best-evidenced item in the file — five studies, two jurisdictions, hundreds of thousands of real decisions.

• The human is the source of the excess disparity. Kentucky bail, 383,080 decisions, 563 real judges, interrupted time series around a reform making non-financial bond the default. Judges deviated from the recommendation 10% more often for moderate-risk Black than for similar moderate-risk white defendants. Post-reform racial disparity was 8.9 pp actual against 4.7 pp that would have obtained under perfect compliance with the algorithm. [preprint — job-market paper, no publication of record] Albright A. If You Give a Judge a Risk Score.

• More algorithm use, more disparity. Virginia felony sentencing, real judges, diff-in-diff plus RD around statewide adoption. No overall incarceration change and no robust recidivism decline; but in the courts where judges used the risk assessment most, Black defendants received ~17% longer sentences than white defendants with similar scores. Judges also granted leniency to the young against their scores — under-23s saw a relative +4 pp incarceration probability. Stevenson MT, Doleac JL. American Economic Journal: Economic Policy. DOI

• Professionals do not filter discriminatory advice at all. 1,411 real HR and banking professionals, Italy and Germany, hiring and lending. Headline: "Human overseers are equally likely to follow advice from a generic AI that is discriminatory as from an AI that is programmed to be fair." The qualitative phase found overseers prioritised organisational interest and had no guidance on when an override was warranted. Gaudeul A et al., European Commission Joint Research Centre, JRC139127, 2025. DOI [peer-reviewed — institutional science-for-policy report, externally reviewed; not a journal article]

• The tool is applied asymmetrically even when identical. 554 crowdworkers, 13,850 predictions on real pretrial defendants. Where the algorithm predicted higher risk than the control group's prediction, it exerted 25.9% stronger influence for Black defendants (0.68 vs 0.54, p = .02). Green B, Chen Y. FAT\ '19*, 90–99. DOI

• The tool silently changes what the decider optimises for. 2,140 participants: the risk assessment made people more responsive to risk and less responsive to competing values. Racial disparity in pretrial detention rose 1.9%; in the loans setting participants became so risk-averse they reduced government aid by 8.3% — a policy shift with no policy decision behind it. Green B, Chen Y. PACM HCI 2021;5(CSCW2):418. DOI

S3 A nominal reviewer is not a safeguard — and the Dutch case proves it structurally

official inquiry · regulator · NGO/advocacy

[official inquiry] [regulator] [NGO/advocacy]

From 2013 the Dutch tax administration ran a self-learning risk classification model over childcare-benefit applications. High scores triggered manual investigation, and selection immediately suspended the benefit.

There was a human reviewer. And — the finding that matters — the reviewing civil servant "did not have access to any details about what information had been used as the basis for assigning a specific risk score." The human step existed; it could not check the machine.

• "Dutch citizenship: yes/no" was a scoring parameter. Non-Dutch applicants were investigated disproportionately.

• Combined with a statutory all-or-nothing rule, any error required repayment of the entire allowance.

• On contesting it: "It was also impossible for parents and caregivers, journalists, politicians, oversight bodies and civil society to obtain meaningful information about the existence and workings of the risk classification model." Parents seeking to know what was wrong "were often met with silence." They "were unable to defend themselves against the system."

• Regulatory findings: the Dutch DPA fined the tax administration €2.75 million (Dec 2021) for three unlawful processings including using nationality as a risk indicator in automated systems — "It is unlawful, and therefore prohibited, to use nationality data to assess applications, combat fraud and determine risk." A second fine of €3.7 million (Apr 2022, its largest ever) concerned a fraud blacklist holding ~270,000 people with no legal basis.

• Scale: tens of thousands of parents; ~42,000 had come forward by June 2021. The Rutte III cabinet resigned on 15 January 2021.

Amnesty International, *Xenophobic Machines*, EUR 35/4686/2021, October 2021 [NGO/advocacy, heavily sourced to the official inquiries] · Autoriteit Persoonsgegevens €2.75m · €3.7m

A widely-cited figure of 1,675 children removed from parental custody comes from Statistics Netherlands (11 May 2022), which called it "an approximation" derived by data matching and noted it does not hold data on court-ordered placements. It is neither a complete count nor a causal finding. Higher figures circulating in press are unverified.

S4 A token review buys as much legitimacy as a real one

peer-reviewed

[peer-reviewed] This is the legitimation critique, measured — and it is the most quotable finding here.

Two preregistered vignette experiments, N = 4,817 and N = 2,834 (Black respondents oversampled to 49.8%), on sentencing, bail and consumer arbitration.

• People rate purely algorithmic decisions as significantly less fair than human ones (p < .001).

• Hybrid human-plus-algorithm procedures close that gap — and "Hybrid Low" (brief review) was rated equivalently to "Human High" (thorough review).

• Not evenly: hybrids fully restored perceived fairness for white and conservative respondents but only partially for Black liberals (Black vs white participants, 4.35 vs 5.02, p < .001).

adding a human restores the appearance of fairness whether or not the review was substantive. That is why "we keep a human in the loop" cannot be accepted as evidence of anything — it is the one claim that pays off identically when true and when hollow.

Chen BM, Hermstrüwer Y, Langenbach P, Stremitzer A, Tobia K. Mitigating the judicial human–AI fairness gap. Journal of Legal Analysis 2026;18(1):207–242. DOI

S5 The policy instrument itself has no empirical foundation

peer-reviewed

[peer-reviewed — law/tech journal]

A survey of 41 policy documents — 23 proposed or passed statutes, 8 guidance documents, 10 relating to contested risk-assessment tools — across the EU, member states, Canada, Québec, Washington, New Jersey, Argentina, Brazil, Australia and elsewhere. Two findings:

"Human oversight policies are not supported by empirical evidence: the vast majority of research suggests that people cannot reliably perform any of the desired oversight functions."

"Rather than protect against the potential harms of algorithmic decision-making in government, human oversight policies create a regulatory loophole: it provides a false sense of security in adopting algorithms…" They "provide the appearance of quality control — legitimizing the use of flawed and controversial algorithms."

Green B. The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review 2022;45:105681. DOI · open: arXiv:2109.05067

Green's empirical foundation leans heavily on his own crowdworker studies, and the professional-population literature does not show the deference his argument assumes (see C1). His conclusion survives; one of his two premises is shakier than the paper implies.

S6 Automated credibility assessment fails on four independent grounds

peer-reviewed · official inquiry

[peer-reviewed] [official inquiry] The strongest-evidenced part of the whole file. "Credibility" is the easiest item on the non-decision list to defend.

The human baseline. Meta-analysis of 206 documents, 24,483 judges: 54% accuracy against a 50% chance baseline — 47% of lies caught, 61% of truths correctly called, with a systematic truth bias. Bond CF Jr, DePaulo BM. Personality and Social Psychology Review 2006;10(3):214–234. PMID 16859438 · DOI

From the National Academies' polygraph review:

"Polygraph screening protocols that can identify a large fraction of serious security violators can be expected to incorrectly implicate at least hundreds, and perhaps thousands, of innocent employees for each spy or other serious security violator correctly identified."

"Its accuracy in distinguishing actual or potential security violators from innocent test takers is insufficient to justify reliance on its use in employee security screening in federal agencies."

National Research Council. *The Polygraph and Lie Detection*. National Academies Press, 2003. DOI

That is a property of screening arithmetic, not of the instrument. It transfers without modification to any automated credibility tool applied across a claimant population where deliberate deception is rare. A critical academic analysis of the EU's iBorderCtrl made the same demonstration concretely: at a conservative 5% base rate, 86.31% of positives would be false — "even if the model did work as reported… the foundational premise of mass screening lacks statistical soundness." That system's accuracy claims rested on 32 actors in a laboratory answering 13 questions, with standard deviations of 24 and 34 points on means of 74–76%. Sánchez-Monedero J, Dencik L. Information, Communication & Society 2022;25(3). DOI · arXiv

And the affect premise is separately unsound. A commissioned consensus review found no one-to-one mapping between facial configuration and emotion, limited reliability, and limited generalisability across cultures and situations; "current algorithms are not accurate enough in their detection of facial [action units] to fully substitute for expert coders." Barrett LF, Adolphs R, Marsella S, Martinez AM, Pollak SD. Psychological Science in the Public Interest 2019;20(1):1–68. PMID 31313636 · DOI

The disability-specific implication — it is an inference, not a measurement. If expression-to-emotion mapping is unreliable in the general population, it is worse for people whose facial movement, affect, eye contact, response latency or prosody differs by reason of autism, cerebral palsy, Parkinson's, facial paralysis, stroke, medication effects, depression, chronic pain, or Deaf communication norms. The populations a disability system assesses are the populations for which the affective signal is least interpretable. No study measures this disparity directly.

S7 What actually binds whom in Canada — and the answer is the duty to accommodate

standard · statute · case law · regulator

[standard] [statute] [case law] [regulator]

everything AI-specific in Canada is either voluntary, federal-public-sector-only, Quebec-only and gated on the decision being exclusively automated, or Ontario-hiring-only. The instrument that actually reaches this work is the one that was already there.

#### Grismer

*British Columbia (Superintendent of Motor Vehicles) v. BCCHR* ("Grismer"), [1999] 3 SCR 868, McLachlin J. for a unanimous Court. Read at source, SCC

Para 19:

"Incorporating accommodation into the standard itself ensures that each person is assessed according to her or his own personal abilities, instead of being judged against presumed group characteristics. Such characteristics are frequently based on bias and historical prejudice and cannot form the basis of reasonably necessary standards."

A predictive model is presumed group characteristics, by construction. That is what a model is — an inference about this person from the pattern of others. Para 19 is not about AI and does not mention it, and that is exactly why it is useful.

Para 22 — "Failure to accommodate may be established by evidence of arbitrariness in setting the standard, by an unreasonable refusal to provide individual assessment, or perhaps in some other way."

Para 42 — "Content to rely on the general opinion of the medical community, and ignoring the evidence that some people with H.H. can and do drive safely, he offered not so much as a gesture in the direction of accommodation." And: "The Superintendent was obliged to give Mr. Grismer the opportunity to prove whether or not he could drive safely, by assessing Mr. Grismer individually."

Para 44 — "It merely establishes that he had a right to be assessed."

Para 41, on cost, which travels straight to the model is cheaper than an adjudicator — "one must be wary of putting too low a value on accommodating the disabled… impressionistic evidence of increased expense will not generally suffice."

#### Meiorin — and the passage that forecloses the obvious defence

*BC (PSERC) v. BCGSEU* ("Meiorin"), [1999] 3 SCR 3, McLachlin J., unanimous. SCC

Para 82:

"individual testing, without more, does not negate discrimination. The individual must be tested against a realistic standard that reflects his or her capacities and potential contributions."

This closes the defence that "we run every claimant through the model individually." Running each person through the same standard is not individualised assessment if the standard itself is not defensible.

Para 65(a) and (c) ask whether the employer investigated "individual testing against a more individually sensitive standard" and whether "standards reflective of group or individual differences and capabilities" could be established. Para 68: "The standard itself is required to provide for individual accommodation, if reasonably possible."

#### CAN-ASC-6.2:2025 — verified clause by clause, full text read

Published December 2025 (the 3 December date is from the news release, not the document). ISBN 978-0-660-79223-1. Full text

Clause 7.1 defines shall = requirement, should = recommendation, may = option, and states that notes contain no requirements.

• 11.4.3 "Consent, choice, and recourse" — the subclause structure is real. Mechanisms shall enable clients, employees and other impacted individuals "to request an equivalently full-featured and timely alternative decision-making process that is, at the individual's choice, either performed: without the use of AI; or made using AI with direct human oversight and verification of the decision." People shall be given accessible-format information on how to correct, contest, change or reverse an AI-assisted decision.

• 11.2.6 "Mitigating statistical discrimination" — organizations shall not subject people with disabilities to AI-assisted decisions "without their informed consent, understanding and access to equivalent alternatives." Tripartite — consent alone is not enough. Its rationale is worth quoting: "Even with full proportional representation in the data, people with disabilities will likely remain outliers or marginalized minorities."

• 11.4.5 — organizations shall ensure "a traceable chain of human responsibility that makes it clear who is accountable."

• 12.11 — alternatives shall offer "equivalent availability, reasonable timeliness, cost and convenience" — the qualifier on timeliness is reasonable, not equivalent. Includes the option of "human supervision or final determination… by a human with expertise in the needs of people with disabilities", and the organization shall retain individuals with that expertise where AI replaces decisions previously made by humans.

• 12.10 transparency, 12.12 feedback/complaints/redress (and it names PIPEDA and provincial privacy legislation — the standard anticipates private-sector users), 10.2.1 documentation published before deployment, 11.1.1 shall avoid "biased proxies… biased data labels… or synthetic data that fails to reflect actual disability experiences."

• 11.2.5 is arguably stronger than 11.1.1: eight named bias sources, prefaced "Care must be taken to recognize that people who are discriminated against in AI-assisted decision making are often people with disabilities." (Drafting note: "must" is not one of the three defined terms in 7.1 — its weight is undefined by the standard's own terminology.)

Status, in the standard's own words: "This document was developed as a reference document for voluntary use." And: Accessibility Standards Canada "has no power, nor does it undertake, to enforce conformance." It is a National Standard of Canada that can be recommended to the Minister responsible for the Accessible Canada Act — and has not been. Accurate framing: a voluntary National Standard, not adopted in any regulation.

C1 "Humans rubber-stamp the algorithm" — not supported

peer-reviewed · NGO report

[peer-reviewed]

• Deployed override rates are 40–60%, not near zero. Virginia judges followed the diversion recommendation in 44% of eligible cases. Kentucky judges' override ran ~50–60%. Allegheny call screeners screened in only 66% of mandatory screen-in cases — overriding against institutional friction. Real admissions officers, in a randomised field experiment over 19,545 applications, were essentially unmoved by which of two disagreeing model scores they were shown (0.42 pp, SE 0.33, p = 0.21).

• Professionals show algorithm aversion, not automation bias. One of six experiments contrasted 301 crowdworkers with 70 national security professionals (Experiment 4): laypeople showed algorithm appreciation; experts heavily discounted all advice including algorithmic, and were less accurate for it (interaction F(1,338) = 5.05, p = .025). Logg JM, Minson JA, Moore DA. OBHDP 2019;151:90–103.

• The one canonical rubber-stamp case is a specific institutional design, not a cognitive law. Polish employment-services profiling: caseworkers overrode in 0.58% of cases — because the override power "was not grounded in legal regulations" and was "treated as exceptional." [NGO report] Panoptykon Foundation, 2015; figure verified via a secondary write-up, not the report PDF.

The correct claim: rubber-stamping is manufactured — by accountability asymmetry (punished for overriding, never for deferring), by surveillance of deviation, by absent legal basis for override, and by throughput pressure. Direct observation of child-welfare screeners found exactly this: pressure to agree with high-risk scores "someone's watching to make sure that the majority of those high risk protocols are being assigned", and override narratives met with responses perceived as "dismissive" or "hostile." Kawakami A et al. CHI '22. DOI

C2 Alon-Barkat & Busuioc finding

peer-reviewed

[peer-reviewed] Three corrections.

Three preregistered experiments, Netherlands. Only Study 3 used civil servants (n = 1,345); Studies 1 and 2 were citizen panels (605, 904). Total N = 2,854, two-thirds laypeople.

1. Automation bias: null in the panels, and reversed in the professionals. Studies 1+2 pooled: OR = 1.07, p = .683. Study 3, civil servants: OR = 0.54, p = 0.006 — professionals were significantly less likely to follow algorithmic than human advice.

2. Selective adherence was marginal and in the wrong population. Study 2 (citizens): OR = 1.50, p = 0.083 two-sided (the paper reports the one-sided 0.042). Study 3 reversed it significantly: OR = 0.57, p = 0.020 — professionals were less likely to sanction the minority-named teacher. The authors attribute this to fieldwork coinciding with the Dutch benefits scandal.

3. It was never shown to be an algorithmic phenomenon. The Algorithm × Ethnicity interaction was non-significant in every study. The authors: "our findings do not suggest that this bias is increased when the score is produced by an algorithm compared to human advice."

Alon-Barkat S, Busuioc M. Journal of Public Administration Research and Theory 2023;33(1):153–169. DOI

The correct statement is narrower and still useful: stereotype-congruent selective adherence to advice exists, and making the advice algorithmic neither creates it nor fixes it.

C3 "Human oversight never works" — too strong, and there is good contrary evidence

peer-reviewed · working paper

[peer-reviewed] + [working paper]

• Real professionals did catch algorithmic errors. A technical glitch caused displayed child-welfare risk scores to diverge from correctly-computed ones. Workers detected it: where displayed scores underestimated true risk, screen-in rose to ~60% against ~30% for correctly-scored cases. 27,575 referral records. De-Arteaga M, Fogliato R, Chouldechova A. CHI '20. DOI

• Some overseers are genuinely good, and identifiable. 37,855 cases, 62 real bail judges: 90% underperformed the algorithm when overriding and 69% did no better than random — but 10% outperformed it, and the only significant predictor was not having prior law-enforcement experience. Angelova V, Dobbie W, Yang CS. NBER WP 31747. [working paper — accepted at ReStud]

• A deployed system reduced racial disparity substantially. Allegheny Family Screening Tool, 60,000+ referrals, triple-differences, among screened-in referrals: case-opening racial gap −5.48 pp (87% of the pre-existing gap); three-month removals −2.87 pp (76%). Note the mechanism: the reduction came from the algorithm constraining human discretion, not from humans correcting the algorithm. Rittenhouse K, Putnam-Hornstein E, Vaithianathan R. Journal of Policy Analysis and Management 2026;45(4). DOI [peer-reviewed — figures from the 2022 working paper]

• A genuine RCT found null. Dane County, 1,891 cases, real judges, randomised: little effect on decisions or on racial differences. Imai K et al. JRSS Series A 2023;186(2):167–189. DOI

C4 "The law requires a human decision-maker" — not in Canada

statute · standard

[statute] [standard]

The right to a human decision-maker is in CAN-ASC-6.2 clause 11.4.3, which is voluntary and adopted in no regulation. The Treasury Board Directive does require a human final decision — at Level III–IV only, and only for federal institutions; Levels I and II expressly permit decisions "without direct human involvement." Quebec s. 12.1 confers a right to submit observations to a member of the personnel of the enterprise who is in a position to review the decision — not a right to a non-automated decision — and only where the decision is "based exclusively on an automated processing" of personal information. The Ontario ESA amendment is hiring-only and disclosure-only.

C5 Two figures in circulation that are wrong or misattributed

official inquiry · press

[official inquiry] [press]

• The Michigan "93% wrong" figure is routinely attributed to the Michigan Auditor General. It is not. The actual OAG performance audit (Report 641-0593-15, February 2016) covers security and claim-processing controls and contains no fraud-determination error rate. The 93% comes from the agency's own review, with the US Department of Labor, of 22,427 determinations, of which 93% were overturned; 2,571 people refunded $5.4 million. Note the denominator — roughly 60,000 determinations were made; ~30,000 were never reviewed.

• the Arkansas coding-error mechanism. A widely-repeated account attributes it to diabetes and cerebral palsy variables. That traces to a 2018 press investigation that could not be retrieved; the lead litigator's own report names conditions as examples but does not attribute the error to a specific variable. What is established: the system had coding errors depriving people of hours beyond the intended cuts, and the state ran no meaningful impact testing before deployment.

N1 No Canadian case, tribunal decision or regulator finding — on any of this [no evidence]

no evidence · case law — US

SEARCH RECORD — 13 September 2026. CanLII full text, all jurisdictions, all courts and tribunals, any date. Query:

(automated OR algorithmic OR "automated decision-making" OR algorithm*) /p ("disability benefit*" OR insurance OR "insurance claim*" OR accommodation)

25 results. Highest relevance 2 of 5; 21 scored 0. Nothing on point. The two strongest hits are access-to-information orders about insurance rating algorithms — Ontario (Insurance Commission) (Re), 1998 CanLII 14216 (ON IPC) and Financial Services Commission of Ontario (Re), 2008 CanLII 68880 (ON IPC) — both about premium calculation and third-party confidentiality, not about a benefits, claims or accommodation decision. One BC WCAT decision mentions impairment-rating software in passing; the rest do not engage automation at all.

What this licenses. The claim can now be stated as a checked negative with its method attached: a CanLII search of all Canadian courts and tribunals, run 13 September 2026, returned no decision in which an automated or algorithmic system's role in a disability, benefits or accommodation decision was at issue.

No Canadian case, tribunal decision, or regulator finding has been identified on automated or algorithmic decision-making in disability benefits, insurance claims, or accommodation. Checked: the Canadian AI Case Tracker (contents are copyright, hallucinated-citation, deepfake, scraping and chatbot-liability matters), OPC investigation findings, and the CHRT — whose only AI instrument is a practice direction on parties' use of AI in proceedings.

The nearest case anywhere is American and worth naming because it is not Canadian: *Kisting-Leung v. Cigna Corp.* (E.D. Cal. 2025) [case law — US], where participants alleged that medical-necessity determinations that policies said would be made by a medical director were in fact made by an algorithm, and fiduciary-breach claims were allowed to proceed.

N2 No peer-reviewed study of proxy divergence in a benefits or work-capacity system [no evidence]

no evidence · peer-reviewed / NGO analysis · official — government self-assessment · press

The mechanism is established — but not here.

The canonical demonstration is a deployed population-health algorithm that predicted future health-care cost as a stand-in for health need. Because less is spent on equally-sick Black patients, cost systematically under-measured their need. At the auto-enrolment threshold only 17.7% of patients flagged were Black; re-specifying the label raised it to 46.5%. At a given risk score, Black patients had ~26% more chronic conditions. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Science 2019;366(6464):447–453. PMID 31649194 · DOI Verified against the PubMed record; the 17.7 → 46.5% figures are in the abstract.

Why it transfers: the error is not about race. It is that a proxy measuring system contact rather than need will under-serve anyone the system already under-serves. Substitute claim duration, service utilisation, treatment cost, or prior benefit expenditure and the mechanism is identical.

The purest documented instance of the disability version is a child-welfare tool that used county behavioural-health service contact — including for conditions such as ADHD — as a predictor, so that using disability services raised a family's risk score. Analysts: these predictors "have the effect of casting permanent suspicion and offer no means of recourse…" [peer-reviewed / NGO analysis] Gerchick M et al. FAccT '23; HRDAG summary. Full text could not be retrieved and the disability-specific quantitative findings were not extracted.

In a return-to-work or accommodation model, "amount of treatment received", "number of specialist contacts", or "prior accommodation requests" would function identically. It is an inference from a well-established mechanism.

And the closest real-world instance is a government that did not measure it. The UK DWP's fairness assessment of an ML model scoring Universal Credit advance requests analysed only age as a protected characteristic — disability was not analysed as a protected characteristic; "reported illness" was analysed as a non-protected one. Claimants reporting illness were 0.37× as likely to be referred and, when referred, correctly identified as fraudulent 0.55× as often. DWP's justification rested on "There is always a human intervention and decision." An earlier analysis obtained under FOI reportedly found significant disparities by age, disability, marital status and nationality — with the figures redacted. [official — government self-assessment] + [press]

N3 Why disability is structurally hard for ML — established, but as argument, not measurement

preprint · institute report · peer-reviewed

[preprint] [institute report] [peer-reviewed]

Three independent sources agree, and the two that most resist rebuttal are arguments about the form of machine learning rather than about any dataset:

• Disability is not a categorical variable, and disabled people are statistically outliers — which ML discards as noise. And a catch-22: group-fairness metrics need the attribute; the attribute is withheld precisely because of discrimination risk. (Exactly what the DWP assessment demonstrates.) Trewin S, arXiv:1811.10670.

• More data makes it worse, not better. "Adding more data to an AI system doesn't address the issue — it simply reinforces the normative model at the core…" Those outside the statistical centre become more remote outliers as the model improves. The illustration: shown footage of a person propelling a wheelchair backwards, successive model iterations chose to run her over with greater confidence. Whittaker M et al., Disability, Bias and AI, AI Now Institute, 2019. [institute report]

• A systematic review (PRISMA, 423 screened → 76 retained → 175 harm instances) found "service/benefit loss" the single most prevalent harm subcategory (n = 28 across 17 papers), and named three harm patterns novel to disability — including inability to verify algorithmic output. Wang L, Kameswaran V, Kacorri H. AIES 2025, 2649–2665. [peer-reviewed]

N4 Disability bias in NLP is established — but the harms measured are the wrong ones

peer-reviewed

[peer-reviewed]

All models tested in one study showed statistically significant explicit bias against disability mentions — inserting a disability mention into a neutral sentence lowered sentiment and raised toxicity scores. Venkit PN, Srinath M, Wilson S. TrustNLP 2023, 26–34.

But the finding that matters most: 56 adults with disabilities across 19 focus groups found the standard evaluation vocabulary insufficient. They rarely described harmful outputs as "offensive" or "toxic" — the categories toxicity classifiers measure. They used "simplistic," "extreme," "dehumanizing," "violation of autonomy," "assumptive," "ableist," and proposed 33 alternative criteria. Gadiraju V et al. FAccT '23, 205–216. DOI

A model can pass every toxicity benchmark and still produce output a disabled person experiences as dehumanising. Any vendor assurance that a system was "tested for bias" is, on this evidence, an assurance about the wrong construct.

N5 No tested oversight intervention has a measured effect in a real deployment [no evidence]

no evidence

Interventions with real measured effects exist — but essentially all of them are crowdworker studies on artificial tasks.

• Best supported: force an independent judgement before revealing the algorithm's. Pretrial gain 0.603 vs 0.464 for seeing the score first; racial disparity in algorithm influence 0.07 vs 0.38 — an 81.5% reduction. Green & Chen, CSCW '19 — crowdworkers. And the one professional test points the other way: 19 veterinary radiologists who judged first became less likely to agree with the AI whether or not it was right, revising only 5% of disagreements. The intervention may convert over-reliance into under-reliance in experts.

• Deliberation time: disagreement with a wrong AI rose from ~48% at 10s to ~67% at 25s — but the deployable version (allocating time by model confidence) on its own "did not lead to sufficient adjustment away from the AI when it was incorrect."

• Explanations do not work. ~1,626 crowdworkers, three tasks: no accuracy gain over showing confidence alone (89% vs 88%, p = .24; 87% vs 86%, p = .22; 67% vs 67%, p = .64) — and explanations increased acceptance of incorrect recommendations too. A PRISMA review of 35 studies (19,774 participants) agrees, adding that explanations can reinforce misplaced trust.

• Outcome feedback is actively harmful — the worst-performing condition tested.

• Cognitive forcing functions work but nobody wants them: overreliance 48% vs 64% (p = .003), and "Participants preferred and trusted more the systems that they perceived as less mentally demanding, but with which they performed poorly."

no intervention in this literature has a measured effect size in a deployed high-stakes setting with real professionals.

N6 Selective adherence in professionals is unreplicated [no evidence]

no evidence

One marginal result in a lay panel, reversed in the one professional sample, with consistent nulls on the algorithm-specific interaction.