What We Learned Testing Jev on Security Alert Triage

Yaniv Zimmer is an AI researcher at Torq, focusing on cybersecurity research at the crossroads of artificial intelligence, deep learning, and defensive operations. Drawing on extensive experience within Unit 8200 alongside industry and academic AI research roles, his work centers on advancing SOC AI architectures and threat detection capabilities

Like a lot of people, we were excited when Jev launched. The premise is genuinely useful in production: an intelligent language model built for classification.

One caveat before the results. Jev is a general-purpose classifier, and security alert triage is a narrow, unusually unforgiving task. We tested it on something it wasn’t specifically built for, and we think the findings say more about the limits of zero-shot classification in this domain than about Jev itself. 

We also ran into the problem the Jev team describes. Consistency is one of the biggest caveats in deployed LLMs, and in security alert triage, consistency is the whole game. When we tested frontier models on triage, the best of them classified consistently only 87% of the time. Run the same alert 10 times, and you will almost certainly get a miss.

Before Jev, we solved that with BERT encoder fine-tuning plus a few additional steps, which we call Torq Reflex. So when Jev was announced, we had an obvious question: Was that work unnecessary? Could we just use Jev instead?

We tested it. The answer is no, and the reason is worth sharing.

Prompt Sensitivity Is the Blocker

We saw that Jev is sensitive to its prompt. An instruction such as “label alerts as malicious only if you see concrete evidence” moved Jev from classifying nearly every sample as malicious to classifying almost none of them that way. For several naive prompts, it collapsed entirely and labeled every alert as a single class, which is a complete failure for a triage task.

That’s the gap between a model that performs well in a general setting and one you can put in front of a production alert queue. When output swings on a sentence of phrasing, the prompt is doing the deciding.

The Numbers

These are preliminary results on a sample of the open-source GUIDE-Microsoft IDS dataset — most specifically, randomly selected tenant n9, and a train:test 2:1 ratio for reflex, sampled with random seed=42.

We see the same trend measured on the full cohort, as well as on high-confidence alerts, where the cohort is determined by the model itself (top 80%).

Jev clearly improves on the earlier ModernBERT approach at the same threshold. But it falls short of the reasoning model (Gemini), and it falls well short of Torq Reflex.

It also scored below what you’d get by simply choosing the majority class. We read that as a consequence of the setup rather than a flaw in the model: without exposure to the data distribution, it has no learned priors for a domain where the priors carry most of the signal.

How We Queried Jev

The Jev query has two parts. The instructions and criteria contain the prompt and the classification options. The state holds the specific alert we’re asking the model to classify.

{
 "state": "GUIDE CLOSED-INCIDENT EVIDENCE ... (truncated)",
  "model": "Jev-latest",
  "questions": {
    "incident_grade": {
      "type": "choice",
      "instructions": "Act as a Tier-3 security analyst performing final incident triage. Classify the described behavior, not the mere existence of an alert...",
      "criteria": {
        "False Positive": "No concrete security-relevant behavior is described...",
        "True Positive - Benign": "A real activity is described and its behavior is best explained as authorized administration, expected software or user activity, testing, simulation, or policy-only activity...",
        "True Positive - Malicious": "Concrete supplied behavior or observables support attack activity, unauthorized access, compromise, harm, hostile infrastructure, or another adversarial explanation."
      }
    }
}

[Full prompt variants and the complete state payload are in the appendix, so results can be reproduced.]

Where Domain Priors Matter

Prompt sensitivity is one limitation. The other shows up when the evidence and the framing disagree.

Here’s an alert Jev correctly classified as malicious:

Alert: Suspicious PowerShell execution
Severity: High
Category: Execution
Description: Encoded PowerShell launched by a user process
Techniques: T1059.001 PowerShell
Obs: command=powershell.exe -enc <redacted> | parent=winword.exe

And here’s a near-identical alert, where only the title changed, which it also classified as malicious:

Alert: Legitimate PowerShell execution
Severity: Low
Category: Execution
Description: Encoded PowerShell launched by a user process
Techniques: T1059.001 PowerShell
Obs: command=powershell.exe -enc <redacted> | parent=winword.exe

The observables are the same in both cases. What changed was the framing, and the classification followed the framing rather than the evidence. A trained analyst weighs winword.exe spawning encoded PowerShell against what’s normal in that environment. A zero-shot model with no exposure to that environment has nothing to weigh it against, which is a property of the approach rather than a bug in any one model.

The Honest Pros and Cons

Jev has real advantages. The context window is larger than the 8K we’re working with in the encoder approach, and you don’t have to retrain when you introduce new labels, which matters if your taxonomy changes often.

The tradeoff is the one that decides it for security: you can’t fine-tune it to your environment or your analysts’ judgment. In triage, that judgment is the product. The knowledge that makes a verdict correct lives in your history and your team’s past decisions, and no prompt can substitute for it.

Why We Still Train

The era of decision-making models is here, and the idea behind Jev is a good one. We’ve been running a version of the same thesis with Torq Reflex, and these results match what that experience taught us: in a domain this specialized, the priors have to come from training rather than from instructions.

If you’re evaluating a platform that claims its AI learns, the question worth asking is whether it trains on your data or rewrites its prompt. Those produce very different results on day 100.

We’d genuinely like to compare notes with the Jev team and others working on this problem, and we’d be glad to see what a domain-adapted version looks like. These results are preliminary, the dataset is open, and we’re happy to share our full prompt set so anyone can reproduce or challenge them.