Does Jev read Hebrew? I tested it on my own tweets
I asked TypeSafe's Jev the same two questions about 9 of my Hebrew tweets and about my English translation of each. It gave the same answer in both languages on 8 of 9. Where it missed, it usually missed in both. A small probe, not a benchmark.
Liran Tal asked on X whether Jev, TypeSafe's small decision model, works on Hebrew. I did not know, so I tested it on the Hebrew text closest to hand: my own last 9 tweets.
The short answer
Yes. On 8 of 9 tweets Jev gave the same answer in Hebrew as on my English translation. When it got something wrong, it usually got it wrong in both languages. Every answer is on this page.
How the test worked
- 9 public Hebrew posts, mostly short quote-tweets, each translated to English by me.
- Two requests per tweet, one Hebrew and one English, both with the quoted post as context. 18 calls on jev-1.13.0.
- Two questions per request: stance, a Choice over six options, and sarcastic, a Noul.
- Hebrew compared with English, and both compared with my own label.
What came back
- Stance: same pick in both languages on 8 of 9. Sarcasm: same side of 0.5 on 7 of 9.
- Against my own labels: 4 of 9 in Hebrew, 5 of 9 in English. About 0.3 seconds per call.
- Idioms cost something: "איזה סרט" ("what a drama") scored 0.70 sarcastic in Hebrew and 0.14 in English. The two answers Hebrew changed came back with low confidence (0.35 and 0.54).
What this is not
Nine tweets, one labeller, my own translations. A probe, not a benchmark. For Hebrew documents, test on your own data and gate on the confidence.
How the page was built
Claude Code ran the test and built the page from an approved text mockup (glyph). Three 21st.dev components ported to plain JavaScript, diagrams in Mermaid, and a browser check with ork:expect from OrchestKit that caught real bugs before publishing.
Want a system like this built for your product?
A free 15-minute intro call. No pitch deck, no pressure. Just whether I can help.