Bilingual Quality Harness

EN / FR output equivalence for a journalistic RAG workflow

Measures whether an AI content pipeline performs equivalently in English and Canadian French — not whether it performs adequately in each. The unit of analysis is the gap between the two, because that gap is what a bilingual public-service mandate actually commits to.

102 parallel queries · 80 parallel documents · Canadian French (Radio-Canada conventions)
mode offline · retrieval embedding · bge-m3
72%
Service parity

6 queries where the French reader was not served — either no answer was returned, or the answer asserted a fact the source does not support. Both are mandate failures, not quality deltas.

Equivalence index 0.968 · 10 dimensions. A mean over all queries understates this: bilingual public service is not an average commitment, so the harness leads with the worst outcome a reader actually experienced.

What the French reader experienced

Every query classified by its worst outcome, ordered by reader impact.

6
Unsupported fact
6
Different sources
17
Reduced content
3
Register
Unsupported fact q03
French asserts unsupported figure(s): 32
EN Environment Canada measured 24 millimetres of ice accretion at Kemptville during the storm.
FR Environnement Canada a mesuré 32 millimètres d'accumulation de glace à Kemptville pendant la tempête.
Unsupported fact q20
French asserts unsupported figure(s): 5.1
EN Wildfires consumed 4.2 million hectares this season, surpassing the 2023 record.
FR Les feux de forêt ont détruit 5,1 millions d'hectares cette saison, dépassant le record de 2023.
Unsupported fact q41
French asserts unsupported figure(s): 34
EN Small business insolvency filings rose 26 per cent, the highest level since 2011.
FR Les dossiers d'insolvabilité de petites entreprises ont augmenté de 34 pour cent, leur plus haut niveau depuis 2011.
Unsupported fact q64
French asserts unsupported figure(s): 145
EN Ambulance offload delays reached a record 118,000 hours, with a median of 92 minutes against a 30-minute target.
FR Les délais de transfert des ambulances ont atteint un sommet de 118 000 heures, avec une médiane de 145 minutes contre une cible de 30 minutes.

Quality dimensions

Each row is one metric measured independently in both languages. The connecting line is the equivalence gap.

English Canadian French — scale 0 to 1, higher is better
Dimension0 → 1EN FRGapStatus
Retrieval precision@1 Top result is a gold document 0.961 0.971 -0.010 equivalent
Retrieval recall@3 Gold documents found in top 3 0.958 0.955 +0.003 equivalent
Retrieval nDCG@3 Rank quality of gold documents 0.954 0.959 -0.005 equivalent
Grounding — fabrication Figures absent from the source document 1.000 0.962 +0.038 equivalent
Grounding — in retrieved context Figures unsupported by what retrieval returned 0.954 0.933 +0.021 equivalent
Grounding — lexical entailment Content words traceable to source 0.915 0.903 +0.012 equivalent
Content coverage vs EN Does FR carry the same facts as EN 1.000 0.884 +0.116 divergent
Canadian French register Metropolitan forms and anglicisms 1.000 0.941 +0.059 equivalent
Fluency (composite) Register and readability combined 0.996 0.955 +0.042 equivalent
Localization conventions Dates, currency, decimals, units 0.990 0.976 +0.015 equivalent

Root cause — lexicon coverage

Canadian French administrative and regional vocabulary the retriever represents weakly. This is what converts a red number into a fix.

Term (stemmed)Model coverageAffected queries

Metropolitan drift and false correction

Two Canadian-specific metrics generic French evaluation does not provide.

60%
Metropolitan drift rate
80%
Quebec false correction rate
20
Rephrased — not drift

MDR counts substitutions only: 3 of 5 scored opportunities. 20 cases where the model rephrased around the term are excluded from the denominator entirely — verbosity is not drift, and scoring it either way would bias the rate.

Quebec formReplaced withnWhere
halte-chaleurcentre d'hébergement chauffé1q04
halte-chaleurcentre d'hébergement chauffé1q29
halte-chaleurcentre d'hébergement chauffé1q77

QFCR sends already-correct Canadian French through a naive “corrige ce texte” prompt, so any alteration is a false correction by construction: 12 of 15 valid Quebec forms were altered. A pipeline can generate perfect Canadian French and still ship metropolitan copy because the proofreading step rewrote it.

Valid Quebec formProofreader changed it tonProbe
fin de semaineweek-end1pf01
courrielemail1pf01
halte-chaleurcentre d'hébergement chauffé1pf01
conseiller scolaireadministrateur scolaire1pf02
taxe foncière(removed/rephrased)1pf02
assurance-emploiassurance chômage1pf03
stationnementparking1pf04
banlieusardnavetteur1pf04

Media metadata — images, video, audio

Alt text, captions, credits, and transcripts. The asset is reused across both language sites; the descriptors often are not.

58%
Accessible in French
5
audio · 1
19
image · 7
7
video · 5

13 of 31 assets leave French users without access to content English users can reach — missing or untranslated alt text, or a missing transcript on timed media. Untranslated is worse than missing: it passes a null check in a CMS audit. The other 18 assets carry complete French descriptors.

AssetKindField states ScoreAccessibility
vid-hlth-028-avideoalt: missingcaption: okcredit: oktranscript: missing0.20alt text missing, transcript missing
img-wx-003-aimagealt: missingcaption: okcredit: ok0.31alt text missing
img-muni-020-aimagealt: missingcaption: okcredit: ok0.31alt text missing
img-envr-029-aimagealt: missingcaption: okcredit: ok0.31alt text missing
img-indg-044-aimagealt: missingcaption: okcredit: ok0.31alt text missing
img-cult-019-aimagealt: untranslatedcaption: okcredit: ok0.48alt text untranslated
img-wx-035-aimagealt: untranslatedcaption: okcredit: ok0.48alt text untranslated
img-hlth-053-aimagealt: untranslatedcaption: okcredit: ok0.48alt text untranslated
vid-wx-003-bvideoalt: okcaption: missingcredit: oktranscript: missing0.50transcript missing
vid-trns-022-avideoalt: okcaption: okcredit: oktranscript: missing0.65transcript missing
vid-indg-025-avideoalt: okcaption: okcredit: oktranscript: missing0.65transcript missing
aud-hlth-041-aaudioalt: okcaption: okcredit: oktranscript: missing0.65transcript missing
vid-wx-049-avideoalt: okcaption: okcredit: oktranscript: missing0.65transcript missing

Editorial metadata

Tags and SEO descriptions drive archive retrieval and topic pages — a French story tagged with fewer concepts is less discoverable in French.

23
Tags dropped in FR
6
Missing FR SEO
0.86
Editorial parity

Only documents with a gap are listed; 18 others are complete.

DocumentDeskFR/EN tags SEO descriptionScore
cult-019Arts2/4missing0.45
tech-026Technology2/4missing0.45
edu-034Education2/4missing0.45
muni-042Municipal2/4missing0.45
hlth-053Health2/4missing0.45
wx-003News3/5missing0.49
envr-029News2/4ok0.80
indg-044Indigenous2/4ok0.80
tech-059Technology2/4ok0.80
cult-006Arts2/3ok0.87
trns-022Municipal3/4ok0.90
cult-032Arts3/4ok0.90
cult-047Arts3/4ok0.90
cult-060Arts3/4ok0.90

Terminology and answerability

Consistency and refusal behaviour across the whole run — failures that are invisible when answers are scored one at a time.

57%
FR terminology consistency
100%
FR answer rate
0
Answered in one language only

A pipeline can render the same programme three ways across a run and each answer looks fine alone. It breaks archive search and house style, and is systematically worse in the second language where no single reviewer sees all the output.

ConceptCanonical form Canonical useMentions Replaced with
warming centrehalte-chaleur0%3
employment insuranceassurance-emploi100%1
property taxtaxe foncière100%1
long term caresoins de longue durée100%2

A refusal asserts nothing, so it scores perfectly on grounding and register. Refusing more often in one language is rationing service by language, and conventional quality metrics cannot see it. EN 100% · FR 100%. within expected expansion (FR/EN 0.90).

Where divergence concentrates

An aggregate says whether the pipeline is failing; segmentation says where, which is what routes it to an owner.

By desk

Deskn Div.Unsvd Gap
Municipal22111+0.197
Arts1150+0.182
News1772+0.176
Politics530+0.167
Business1341+0.154
Education940+0.148
Health1241+0.097
Technology1021+0.067

By region

Regionn Div.Unsvd Gap
Ontario23114+0.206
West931+0.167
Quebec1560+0.162
Prairies1470+0.131
Atlantic930+0.093
National3081+0.083
North1450+0.071

By topic

Topicn Div.Unsvd Gap
recycling thin111+1.000
arts_funding330+0.667
broadcasting210+0.500
fire_service thin110+0.500
music210+0.500
school_funding220+0.500
small_business thin111+0.500
snow_removal220+0.500

Per-query detail

Full result table — the accessible view of every number above. Divergent queries first; perfect scores are dimmed so the values that differ carry the eye.

QueryEN answerEN P@1 FR P@1EN grndFR grnd FR regFR cov
q03Unsupported fact Environment Canada measured 24 millimetres of ice accretion at Kemptville during the storm… 1.001.00 1.00 0.00 1.00 0.00
q20Unsupported fact Wildfires consumed 4.2 million hectares this season, surpassing the 2023 record.… 1.001.00 1.00 0.50 1.00 0.50
q41Unsupported fact Small business insolvency filings rose 26 per cent, the highest level since 2011.… 1.001.00 1.00 0.50 1.00 0.50
q64Unsupported fact Ambulance offload delays reached a record 118,000 hours, with a median of 92 minutes again… 1.001.00 1.00 0.67 1.00 0.67
q83Unsupported fact Producer-funded recycling shifted an estimated $156 million in annual costs from municipal… 1.001.00 1.00 0.00 1.00 0.00
q86Unsupported fact All new open datasets require full bilingual metadata from July 1; 41 per cent of existing… 1.001.00 1.00 0.50 1.00 0.50
q15Different sources Federal supports include a $12 million touring fund expansion with a quarter reserved for … 1.001.00 1.00 1.00 1.00 0.67
q29Different sources People can go to warming centres — municipalities opened eleven during the ice storm, and … 1.001.00 1.00 1.00 0.00 0.50
q31Different sources Nursing vacancies drove 1,140 rural emergency room closures, while labour shortages delaye… 0.000.00 1.00 1.00 1.00 0.83
q32Different sources The broadcaster is opening three bureaus in official language minority markets, and the ar… 1.001.00 1.00 1.00 0.00 1.00
q51Different sources Employers cited housing availability as the main barrier, and a federal subsidy now covers… 1.001.00 1.00 1.00 1.00 1.00
q75Different sources Supports include a settlement transferring 78,000 hectares, a country food program in eigh… 0.000.00 1.00 1.00 1.00 1.00
q07Reduced content A quarter of the $12 million touring fund expansion is reserved for official language mino… 1.001.00 1.00 1.00 0.00 0.67
q102Reduced content Eviction applications reached 92,000 with an 8-month median hearing wait, while shelters r… 1.001.00 1.00 1.00 1.00 0.75
q13Reduced content Yes. The department admitted 8.1 per cent of newcomers outside Quebec as French speakers, … 1.001.00 1.00 1.00 1.00 0.50
q14Reduced content School boards face a combined $88 million shortfall amid 5.4 per cent enrolment growth, wh… 1.001.00 1.00 1.00 1.00 0.50
q19Reduced content Rural hospitals reported 1,140 temporary emergency room closures last year, more than doub… 1.001.00 1.00 1.00 1.00 0.83
q22Reduced content Youth unemployment reached 14.2 per cent, the highest since 2016, while the overall rate r… 1.001.00 1.00 1.00 1.00 0.67
q26Reduced content On-time performance was 67 per cent, below the 80 per cent target, largely due to freight … 1.001.00 1.00 1.00 1.00 0.74
q35Reduced content Drought cut the Prairie crop forecast 18 per cent after a dry June, with poor soil moistur… 1.001.00 1.00 1.00 1.00 0.50
q42Reduced content A court ruling on section 23 Charter rights ordered a remedy, and the province will fund t… 1.001.00 1.00 1.00 1.00 0.50
q49Reduced content Drought cut Prairie crop forecasts 18 per cent while a post-tropical storm flooded 620 coa… 1.001.00 1.00 1.00 1.00 0.33
q54Reduced content The city is reviewing snow removal contracts after 4,800 service complaints, with twelve o… 1.001.00 1.00 1.00 1.00 0.33
q62Reduced content Ferry service was cut from ten to six daily crossings after a vessel was removed for hull … 1.001.00 1.00 1.00 1.00 0.67
q71Reduced content The museum returned 1,100 cultural objects over three years, with 62 requests still in pro… 1.001.00 1.00 1.00 1.00 0.67
q73Reduced content The city is reviewing snow removal contracts after 4,800 complaints, extreme cold closed s… 1.001.00 1.00 1.00 1.00 0.67
q82Reduced content The housing guarantee helped fill 92 of 108 vacant teaching positions, with eleven still v… 1.001.00 1.00 1.00 1.00 0.67
q95Reduced content Volunteer fire departments reported a 24 per cent decline in active members, with 41 depar… 1.001.00 1.00 1.00 1.00 0.50
q99Reduced content Nursing overtime fell 38 per cent, dental plan enrolment passed 3.1 million, and youth men… 0.001.00 1.00 1.00 1.00 0.50
q04Register Municipalities opened eleven warming centres across the affected region for residents with… 1.001.00 1.00 1.00 0.00 1.00
q05Register Trustees asked the province for a per-student funding increase of $410, as the four boards… 1.001.00 1.00 1.00 0.00 1.00
q77Register Twenty-two warming locations opened in the city during the extreme cold warning.… 1.001.00 1.00 1.00 0.00 1.00
q01 Seasonal workers in Atlantic Canada now need 420 insurable hours to qualify for employment… 1.001.00 1.00 1.00 1.00 1.00
q02 Quebec signed separately from the other nine provinces, securing an asymmetrical arrangeme… 1.001.00 1.00 1.00 1.00 1.00
q06 The authority deferred the increase after four municipalities objected. The toll was to ri… 1.001.00 1.00 1.00 1.00 1.00
q08 Halifax had the tightest rental vacancy rate at 0.9 per cent, against a national rate of 1… 1.001.00 1.00 1.00 1.00 1.00
q09 Thirty-eight federal judicial appointments remain vacant, with 22 in final review at the j… 1.001.00 1.00 1.00 1.00 1.00
q10 The central bank held its policy rate at 3.75 per cent for a third consecutive time and si… 1.001.00 1.00 1.00 1.00 1.00
q100 Spring melt evacuated 1,240 residents while drought cut the Prairie crop forecast 18 per c… 1.001.00 1.00 1.00 1.00 1.00
q101 New open datasets require bilingual metadata from July 1, and a publishing grant supports … 1.001.00 1.00 1.00 1.00 1.00
q11 The snow crab quota for southern Gulf zones was cut 22 per cent to 18,400 tonnes, affectin… 1.001.00 1.00 1.00 1.00 1.00
q12 The city's $4.1 billion operating budget carries a 3.9 per cent residential property tax i… 1.001.00 1.00 1.00 1.00 1.00
q16 An ice storm cut power to 180,000 homes and caused two deaths on Highway 417, while a plan… 1.001.00 1.00 1.00 1.00 1.00
q17 The national dairy commission reduced production quota by 1.8 per cent, the first cut sinc… 1.001.00 1.00 1.00 1.00 1.00

Divergent queries listed first. 58 further queries of 102 showed no divergence on any dimension and are omitted here; every one is in the JSON output.