Vary one thing at a time
Start with a clear reference question and an approved answer. Create variants using Arabic, English and a natural mixture drawn from de-identified customer language. Keep the facts unchanged so failures can be attributed to wording rather than a different scenario. Have a fluent reviewer check that a dialect variant means what you intended; invented slang is a poor test of real customers.
Score the operational result
Check whether the system retrieved the relevant policy, preserved the customer's identifiers and made the correct decision to answer or hand off. A language label alone is not enough. Record misunderstood terms and compare results by variant. Do not generalize a small Saudi sample into a claim that the product understands all Arabic dialects or every mixed-language input.
Fictional worked example
Fictional pair: 'Can I change the delivery address before dispatch?' and 'أقدر أغير الـ delivery address قبل الشحن؟'. Both should reach the same policy and preserve the condition 'before dispatch.' Add a dispatched-order variant as a separate scenario; if that case receives the same answer, the failure is a condition error rather than merely a translation problem.
Action checklist
- Keep intent and facts fixed across each language pair.
- Use fluent review for locally natural phrasing.
- Score retrieval, factual conditions and handoff separately.
- Report sample size and tested variants with any result.