10 · Jev + Gemini · demo 6CSV evaluation: Jev vs Gemini
CSV rowcomment + expected decision
Jev answers3 questions · probabilities
🤖 Gemini answers3 questions · labels
policy
→ decision
policy
→ decision
vs expectedwho agrees more with the human · how fast
THREE QUESTIONS, ONE OF EACH TYPE
const jevQuestions = {
  category: choice("What kind of comment is this?", categories),
  toxicity: score("How offensive is the tone?", toxicityLevels),
  personal_data: noul("Contains an email, phone number or address"),
};
BOTH MODELS ON EVERY CSV ROW
const [jevResult, geminiResult] = await Promise.all([
  timed(() => jev.systemOne({ state: comment, questions: jevQuestions })),
  timed(() => generateText({ model, prompt: comment, output })),
]);
THE POLICY LIVES IN OUR CODE, NOT IN THE MODEL
const signals = [
  { name: "Spam", risk: a.category.probabilities.spam },
  { name: "Toxicity", risk: a.toxicity.score / 3 },
  { name: "Personal data", risk: a.personal_data.noul },
];
const verdict = (risk) =>
  risk >= reject ? "reject" : risk >= review ? "review" : "publish";
Same policy for both: who agrees with the human?