Audacix

How it was earned

Choosing the model, and why it worked

The assistant ran on Llama 3.1 8B, quantised to 4-bit, on a single 16 GB GPU shared by everyone using it. I moved it to Qwen 2.5 7B at 8-bit, and I chose on practical grounds: enough context for a user's scan, room for several people asking at once at a steady speed, and how each model behaved, reasoned and held its guardrails.

The table below is arithmetic I worked out afterwards. It explains why the choice worked; it isn't how I made it. At full 16-bit precision, none of the candidates loads with room left to answer anyone.

source · weights: published sizes on Hugging Face · room: arithmetic, not measured

The last two rows are why it worked. Qwen at 8-bit has bigger weights than Llama at 4-bit, yet leaves more room for context: its memory per token of context is less than half Llama's, with 4 key-value heads to Llama's 8 and 28 layers to 32. Better precision cost no context.

I stress-tested the ones that fit. The 14B loaded but barely ran. Among the rest I looked at speed, at how well each stayed inside the assistant's scope and followed its output format, and at how it handled the code and reasoning that web-security questions need. On speed the two came out about equal, so the choice came down to behaviour and room. There was no written behaviour benchmark behind it, and next time I'd build that test set first. I made the switch by setting its context window below the maximum to keep memory headroom, and tuned vLLM's settings for latency.

So that speed was measured rather than felt, the assistant logs time to first token and tokens per second on every answer.

Guarding a stream without new compute

The model is treated as untrusted. A jailbreak check runs on every question before anything is generated, and the answer is checked for jailbreaks and for staying on web security before the user sees it.

The catch: the open-source guardrails framework we used checked a finished answer, not a stream. Waiting for the whole answer meant a blank screen, and on the CPU we had, the topic check took tens of seconds per answer. A second GPU for the checks was the obvious fix, and I didn't have one.

So I streamed the answer and checked it in pieces. Text is held back until a few sentences are complete, which gives the small classifier models enough context to judge. Each piece is checked in the background while the next one streams, and the stream stops the moment a piece fails. People see an answer start almost at once. The trade-off was chosen on purpose: a violation can surface a sentence or two in, rather than never being shown at all.

Before release I ran it against more than 140 adversarial prompts, from prompt injection and role-play to emotional manipulation, persuasion and multi-step attacks, and checked each refusal held.

Not reaching for a vector database

The assistant answers questions about this user's scan. That data was already structured in the database, and the general knowledge it needed was small and predictable. So context came straight from the scan records and a small fixed knowledge base. A vector store would have been one more service to run and keep in sync, for a worse answer: similar documents instead of the exact scan.

Debugging past my own component

Users waited minutes for a scan result that never came. The scan wasn't slow; it had never started, because one path marked the request complete without queuing the work. I traced it past my own component and made that path queue the scan and add its result to the report when it finished.

What stands

Still in the live product, although some layouts have changed since:

• the assistant, on the scan results;
• the scan-queuing fix;
• the certification offered to sites that score 4+ out of 7;
• technology detection and JavaScript vulnerability detection;
• security headers and frontend changes.

What this can't show

• The code belongs to Audacix and isn't public, so this page describes decisions, not code. What I could check against the code, I did. The assistant itself anyone can see by running a scan, as in the recording above.
• My stress tests and the model comparison were written up for Audacix and stayed there; what's above is the arithmetic anyone can redo, and my account of the rest.

Sources

• Internal to Audacix
• Model weights: the published file sizes on Hugging Face for each model and precision (Llama 3.1 8B at 8-bit is computed, as 1 byte per weight plus 16-bit embeddings). Room for context: about 12.5 GB usable after runtime overhead, minus the weights, divided by the model's key-value cache per token (Llama 3.1 8B 128 KB, Qwen 2.5 7B 56 KB, Qwen 2.5 14B 192 KB, at 16-bit). It is shared by everyone asking at once.
• The live scanner, where the assistant runs on the scan results ↗