Transcript

A 12 gigabyte graphics card is running a 125 billion parameter AI model. And somehow it' generating around 22 tokens per second. The model takes roughly 68 gigabytes of storage. The computer has just 16 gigabytes of system RAM and the GPU is an ordinary NVIDIA RTX 3060. But there's a catch. The original model supports a context window of 262,144 tokens. This experimental setup is configured for just 512. So no, you aren't about to replace ChatGPT or run a massive coding agent on your old gaming PC. But that's not what makes this interesting. A developer has built a custom version of llama.cpp that keeps parts of this enormous model on an NVMe SSD and brings the weights into GPU memory only when they're needed.

And the really interesting part is that it hasn't achieved this by deleting the model's experts or making a completely different much smaller model. It's changed how the model accesses its data. So how does a 12 gigabyte graphics card run something this large? Where do those 68 gigabytes actually live? And can this technique eventually make expensive GPUs less important for local AI? Let's get into it. First, we need to understand why this particular model is such a good candidate for this experiment. The model is Quinn 3.8 Flash Next and it uses something called a mixture of experts architecture.

It has 125 billion parameters in its main language model, but only around 6 billion are activated for each token. And no, that doesn't mean somebody deleted 119 billion parameters. All those parameters are still there. The model is divided into hundreds of specialized experts. When it processes a token, a routing system selects the experts it needs instead of sending that token through every single expert. There are 512 routed experts in each mixture of experts layer and the model activates 10 routed experts alongside a shared expert. Think of it like a company with hundreds of specialists. You need every employee working on every problem.

You select the people who are relevant to the task. That's the basic idea. But there's a catch with this architecture. Even when an expert isn't being used, its weights still need to exist somewhere. And across 48 layers, we're dealing with 24,576 routed experts. That's a lot of data to squeeze into a 12 GB graphics card. And this is where conventional inference starts running into problems. Normally, when you load a large model, you try to place its weights in GPU memory. When they don't fit, you start offloading parts of the workload to system RAM or storage. But repeatedly retrieving missing weights can destroy performance.

Your GPU can calculate incredibly quickly, but it can't calculate with data that hasn't arrived yet. It's like having a sports car that spends most of its time waiting for somebody to deliver the fuel. You haven't made the car slower, you've made the supply system the bottleneck. And this is exactly the problem Baidu Juice tries to solve. Baidu Juice is a custom fork of llama.cpp built around a technique called demand paging. Instead of requiring every expert to stay inside GPU memory, it maintains a limited pool of available experts and retrieves missing ones when they're required. The idea sounds simple.

The implementation isn't. Let's look at what actually happens when the model generates a token. The dense parts of the model remain resident on the GPU. These are the components needed for the regular computation. The routed experts work differently. Only a limited number of expert weights can occupy the GPU's available slots. When the routing system selects an expert, the engine checks whether that expert is already present. If it is, great. The GPU can use it. If it isn't, the system needs to retrieve it. Baidu Juice uses three levels of storage to manage this. The GPU's VRAM is the fastest workspace.

It holds the dense computation and a limited number of cached experts. System RAM acts as a victim cache. When an expert gets evicted from the GPU's slot pool, the system can retain it in host memory rather than immediately throwing that copy away. And then there's the NVMe SSD. That's where the larger collection of expert weights lives. When a required expert isn't available in either memory cache, the paging system requests it from the SSD. The data is read asynchronously through the host-side pipeline and transferred into an available GPU slot. The important part is that these operations are coordinated instead of making the GPU wait for every storage request to finish before doing anything else.

The system also uses techniques such as CUDA graph replay and a custom GPU-side cache lookup mechanism to reduce the coordination overhead. And this is why Baetalgens is not simply somebody running a normal model through a slow SSD. The model loader, storage format, GPU cache, and paging system have all been designed around the same idea. Keep the data that is needed immediately close to the GPU. Keep recently used data in RAM. Leave the rest on storage until it is needed. And there's one detail I really like. Baetalgens hasn't achieved this by simply throwing away the experts that aren't used very often.

The project preserves all 24,576 routed experts in its packaged model. Their existing quantized weight data is retained. It changes their storage layout so the inference engine can retrieve them individually. That's a very different achievement from pruning a model until it fits. But where does the 68 GB actually come from? The packaged model is called Alpha Orionis XS. It's based on Qwen 3.8 Flash Next and uses the GSQ-RCO-IQ2 XS quantization from ISTA-DASLab. The complete runtime bundle contains four files. The first is the dense GGUF file. This contains the dense weights, routers, metadata, tokenizer, and other components required by the inference engine.

Then there's a file called experts.bin. This is where the routed expert weights are stored separately, organized so the paging engine can retrieve the data it needs. The third file contains the model's per layer n-gram embedding table. This is a particularly large component of Queen's architecture, with approximately 51 billion parameters in the original model. And finally, there's a small profile file used to help initialize the GPU's expert cache. Together, these files make up roughly 68 gigabytes of model data. But there's something important about that quantization. IQ2 XS is an aggressively compressed weight format.

Beetlejuice isn't taking a full precision model and somehow squeezing it into a magical new format. The underlying quantization already exists. The project repackages those weights into a layout that its paging engine understands. It doesn't train a new model, distill it into a smaller model, prune the experts, or requantize the existing weights. That's important because it separates two completely different achievements. The quantization reduces the amount of data needed to represent the model. The paging system changes how that data is accessed. And you need both to make this particular configuration practical.

There's another limitation worth mentioning here. The original Queen model also includes a multi-token prediction component and vision capabilities. The Beetlejuice bundle doesn't include the MTP sidecar or vision projector. So we're talking about a specialized text generation configuration, not every capability of the original Queen model. Now, enough about the architecture. Let's talk about the number that got everyone's attention. 22 tokens per second. According to the developers' published measurements, the warm repeated workload decode result reached approximately 22.03 tokens per second on an RTX 3060 with 12 gigabytes of VRAM.

The system also had 16 gigabytes of DDR4 memory and a consumer NVMe SSD. And there's a useful detail about the testing environment. The process was restricted to a 12 gigabyte memory limit with swap disabled. So this wasn't a case of secretly giving the program another 64 gigabytes of virtual memory and pretending it was running entirely on a normal system. The developer also recorded approximately 12.43 megabytes of physical SSD reads per generated token in the optimized configuration. That's interesting because the complete model bundle is around 68 gigabytes. It isn't reading the entire bundle again for every token.

It's retrieving the specific data that isn't already available in the caches. Now I need to be very clear about the performance claim. The 22.03 figure is a developer reported warm repeated workload measurement. It isn't an independent benchmark and it doesn't mean every prompt will generate at that speed. A different prompt can request different experts. A cold cache can behave differently from a warm one and a longer context can change the workload considerably. The model's generation speed is also not the same as its prompt processing speed or time to first token. So don't take 22 tokens per second and assume your computer will behave exactly the same way.

But even with those qualifications, the result is genuinely interesting. A model this large is being run on a single consumer graphics card with a relatively small amount of system memory. And now we get to the biggest problem. The context window. Remember when we mentioned that Qwen 3.8 supports 262,144 tokens? Well, the documented Beetle 2 setup uses a context size of just 512 tokens. And that changes the entire story. A context window determines how much input and conversation the model can keep available while generating response. With 512 tokens, you have very little room. You can't comfortably paste a large source code file into the prompt.

You can't give it an entire document and ask it to analyze everything. You can't maintain a long conversation where the model needs to remember dozens of previous instructions. And running a coding agent with a tiny context window would be particularly frustrating. The model can generate responses, but the amount of information it can work with is severely restricted. This is important because memory paging and context length solve two different problems. Paging helps the system access the model's weights without keeping all of them in VRAM. Context length determines how much information the model can process in a conversation.

Fixing the first problem does not automatically fix the second. And that's exactly what this demonstration proves. It shows that a very large mixture of experts model can be executed using an unusually small GPU memory budget. It does not demonstrate that every feature of the original model works or that you can run it at its native context length. The current documented configuration also uses one paged context in one server slot. So if you were imagining several people using this as a shared local AI server, that's not what has been demonstrated either. The project itself describes the implementation as experimental and partly broken.

And there are hardware restrictions. The documented configuration targets Linux on x86-64 and NVIDIA RTX 3060 with 12 GB of VRAM and CUDA compute capability SM 86. It also needs 16 GB of system RAM and enough local NVMe storage for the model bundle. This isn't a normal Ollama installation. You need the Bedeljuice fork because its specially packaged model files are not compatible with stock llama.cpp, Ollama, or LM Studio. The project recommends using its server executable, Bedel server, with the documented paging configuration. And if you're using Windows, a Mac, or an AMD graphics card, don't assume you can simply download the files and reproduce the demonstration.

The current published results don't establish that. There's also a practical question about the model itself. We're using an aggressively quantized version of Quen 3.8. Preserving the quantized bytes during repackaging doesn' prove that this version has the same quality as the original BF16 model. Those are separate questions. You would still need proper evaluations of coding, reasoning, instruction following, and long-form output to determine how well this particular quantized build performs. And because the context is so restricted, even an excellent response to a short question doesn't tell us how well it would handle a complex real-world task.

So where does that leave us? Would I replace my normal local AI setup with Beetlejuice? No. The context limitation alone makes that a difficult recommendation for everyday use. The hardware target is narrow, the software is experimental, and the setup requires more work than opening a normal model runner. But I wouldn't dismiss the project either. Because the interesting part isn't that your SSD is suddenly replacing your GPU. It's that we found another way to think about memory. For years, buying local AI hardware has largely meant buying more VRAM. If the model doesn't fit, you need a bigger graphics card.

If you want a larger model, you need another card or a much more expensive workstation. But mixture of experts models give inference engineers another opportunity. Not every expert is needed for every token. Frequently used experts can remain cached. Other experts can live in system memory. And less frequently used data can be retrieved from storage when necessary. That doesn't make storage as fast as VRAM. It doesn't eliminate transfer latency. And it doesn't mean you can run every model this way. But it shows that the software managing your memory can make a very substantial difference. And I think that's the bigger story.

The next generation of local AI performance might not come entirely from buying bigger graphics cards. It could also come from getting much smarter about where model data lives, what needs to be loaded, and when it needs to move. Bedlejuice is an early and very experimental example of that idea. It isn't ready to replace your everyday AI setup, but it demonstrates something worth paying attention to. So tell me what you're running right now. How much VRAM does your GPU have? And what's the biggest model you've managed to run on it? And would you actually try something like Bedlejuice knowing that the current configuration has a 512 token context limit?

I'll catch you in the next one.

Seek backward 5s ←
Longer seek (10s) Shift ←
Play Space Seek forward 5s →
Longer seek (10s) Shift →
0:00 Auto-scroll A Copy link at current time S

This New llama.cpp Fork Turns Your SSD Into VRAM (22 Tok/s)