I built a SOC analyst assistant that runs entirely offline, produces the same output every time you give it the same input, and contains no model weights. It’s a lookup engine with structured logic wrapped around a knowledge base that I built and maintain manually. I call it CARL.
The obvious question is: why not just use an LLM?
It’s a fair question. LLMs are good at open-ended analysis. They synthesize information quickly. They don’t require you to pre-enumerate every case. For a lot of tasks, they’re clearly better than a static lookup engine. I use them. But for the specific use case of “analyst needs consistent, auditable guidance during an active investigation,” they have problems that the architecture I built doesn’t have.
The audit trail problem
In a CMMC-adjacent environment, actions taken during an investigation may need to be reviewed. When an analyst makes a decision — close this alert, escalate that one, apply this playbook — there’s often a downstream question: what guided that decision?
With a deterministic tool, the answer is clear. The analyst queried X, the tool returned Y, the logic that produced Y is versioned and reviewable. You can pull up the exact knowledge entry that drove the recommendation. You can show it to a client, an auditor, or your own management.
With an LLM, the answer is: “The model said so.” You can log the prompt and the response, but the reasoning is not inspectable in the same way. The model might give a different answer tomorrow to the same question. That’s a meaningful difference when you’re trying to demonstrate process consistency.
Reproducibility in a high-volume environment
Shift-to-shift consistency matters in an MSSP. If analyst A works a Password Spray alert on Monday and gets one set of guidance, and analyst B works the same alert type on Wednesday and gets different guidance, you get inconsistent investigation notes, inconsistent close rationale, and inconsistent outcomes for clients.
LLMs don’t guarantee reproducibility. Temperature settings and model updates affect outputs. A deterministic engine with a curated knowledge base gives the same answer to the same query every time. That makes it easier to train new analysts, easier to QA investigations, and easier to identify gaps in the knowledge base — because the gaps are explicit and enumerable, not hidden inside a probability distribution.
The compliance constraint
The environment I work in has strict controls around external data transmission. Running a local LLM is possible but requires infrastructure that isn’t always available. Cloud LLMs require sending investigation context off-premises, which creates data handling questions that aren’t always easy to resolve quickly.
A tool that runs entirely in the browser or as a local process with no external dependencies sidesteps that entirely. There’s nothing to audit from a data egress standpoint because there’s no egress. The knowledge base is a flat file. The logic is JavaScript or Python, depending on the surface. Anyone on the team can inspect it.
What it gives up
I’m not arguing that deterministic tools are better in general. They’re worse in several ways that matter.
They don’t generalize. If an attack technique isn’t in the knowledge base, the tool has nothing to say. LLMs handle novel cases gracefully; a lookup engine returns nothing useful.
They require ongoing maintenance. When TTPs shift, when new alert types appear, when clients add new tooling — someone has to update the knowledge base. That’s manual work, and it falls behind.
They’re brittle on input variation. A well-written LLM prompt handles typos, rephrasing, and ambiguity. A lookup engine needs the query to match the way the knowledge base was built.
The tradeoff is deliberate. For SOC guidance specifically — standardized playbooks, escalation criteria, known-good close logic — determinism and auditability are worth the maintenance cost. For open-ended threat hunting or novel incident analysis, I use different tools.
Build for the use case, not the technology. The instinct to reach for an LLM because it’s capable is understandable. But “capable” and “appropriate” aren’t the same thing.