Repository navigation
evals: add RTL language coverage #197
Description
Add RTL and multilingual evaluation coverage
I think it would be useful to add a few evaluation cases for RTL languages, especially Persian and Arabic.
The current evals do a good job of measuring response structure and actionability, but they are mostly English-focused. Some of the existing rules can behave differently when RTL text is mixed with LTR content such as commands, file paths, URLs, and code.
For example:
src/auth.ts
npm test -- auth.spec.ts
can become harder to scan when embedded in RTL text. Numbered steps and punctuation can also be affected.
Proposal
Add a small set of language-aware cases covering:
enas the controlfa(Persian)ar(Arabic)
The cases could cover:
- action-first responses
- multi-step tasks
- mixed RTL/LTR technical content
- multi-turn state
- verbosity vs. actionability
Something like:
{
"id": "rtl-debug-fa-001",
"language": "fa",
"direction": "rtl",
"rule": "action-first"
}The goal isn't to add language-specific rules to the skill. This should stay in the eval layer and simply verify that the existing behavior holds across different languages and writing directions.
Acceptance criteria
- Add representative Persian cases
- Add representative Arabic cases
- Keep English control cases where appropriate
- Cover mixed RTL/LTR content
- Extend the rubric for RTL/LTR readability
- Document how to run the new cases
- Ensure existing English evals are unaffected
I think this would be a useful extension of the current evaluation system and a good way to catch issues that aren't visible in English-only testing.
Activity