The project in plain English: a hummingbird carrying a whale

A developer going by JustVugg built something that shouldn’t work on paper. The project is called colibrì — Italian for hummingbird — and it runs GLM-5.2, a 744-billion-parameter language model, on a consumer machine with roughly 25GB of RAM. No GPU. No cloud subscription. No Python environment, no CUDA stack, no framework dependencies of any kind. Just a single binary and a hard drive.

The engine is written in pure C, zero external libraries. Type ./coli chat and the model is ready in 32 seconds, holding only 9.9GB resident in RAM at any given moment. That number demands a second look: 9.9GB of RAM to serve a 744-billion-parameter model.

The trick is architectural, and GLM-5.2 makes it possible. The model is a Mixture-of-Experts design, meaning it activates only around 40 billion parameters per token rather than the full 744 billion. JustVugg splits the model’s weight along that boundary. The dense components — attention layers, shared experts, embeddings, roughly 17 billion parameters — stay compressed in RAM at int4 quantization, accounting for that 9.9GB footprint. The routed experts, all 21,504 of them spread across 75 MoE layers with 256 experts each, live on disk. The full disk footprint runs to about 370GB, with each expert weighing roughly 19MB at int4. When the model needs a specific expert during inference, colibrì streams it from disk on demand, managing the flow with a per-layer LRU cache.

The name colibrì earns its irony. A hummingbird skeleton weighs almost nothing; it carries a whale. The local LLM inference engine itself is negligible — a small C binary anyone can compile — but it moves one of the largest open language models available through a machine most people already own.

Local AI inference at this parameter scale was, until very recently, a data center problem. Dedicated GPU clusters, terabytes of high-bandwidth memory, specialized hardware — the on-premise large model deployment assumption ran straight through expensive infrastructure. colibrì punches a hole in that assumption using disk I/O, a smart caching layer, and a quantization format that most laptop SSDs can handle.

The architectural loophole: Mixture-of-Experts is not what most people think

When people hear “744 billion parameters,” they picture a model that needs 744 billion parameters worth of hardware to run. That assumption is wrong, and the architecture of GLM-5.2 is the reason why.

GLM-5.2 is a Mixture-of-Experts model. Rather than one monolithic network where every parameter fires on every input, a MoE architecture divides its weights into specialist sub-networks called experts. A learned routing mechanism reads each incoming token and selects only a small subset of those experts to handle it. The rest of the network sits idle for that token, consuming no compute.

The numbers inside GLM-5.2 make this concrete. The model carries 744 billion total parameters, but only roughly 40 billion activate per token. Of those 40 billion, only about 11 gigabytes worth of weights — the routed experts — actually change from token to token. The remaining active parameters belong to components that stay fixed across the entire conversation: attention layers, shared experts, and embeddings, totaling around 17 billion parameters that fit in roughly 9.9 GB of RAM at int4 quantization.

The full roster of routed experts is enormous — 21,504 of them across 75 MoE layers, each weighing about 19 MB at int4, adding up to around 370 GB on disk. But the router only calls a handful per layer per token, which is why the live memory footprint stays manageable.

This is the distinction mainstream AI coverage consistently flattens: total parameter count measures a model’s capacity, not its per-token computational cost. A dense 40B model activates all 40 billion parameters for every single token. A 744B MoE model activates roughly the same number of parameters per token while carrying far greater total knowledge — the extra parameters represent specialists waiting to be called, not constant overhead.

Sparse model inference, expert routing, and on-demand weight loading collectively produce something that looks paradoxical on a spec sheet: a three-quarter-trillion-parameter language model whose active compute footprint fits inside consumer hardware. The parameter count headline is not a lie, but it describes a ceiling, not a floor.

The engineering insight: stream what you need, forget the rest

GLM-4’s architecture hands Colibrì a structural gift. A Mixture-of-Experts model doesn’t activate all its parameters at once — for each token it processes, GLM-5.2 fires roughly 40 billion of its 744 billion parameters. That selective activation is the mechanical fact the entire engine is built around.

Colibrì splits the model into two categories based on that behavior. The dense components — attention layers, shared experts, embeddings — total about 17 billion parameters and are needed for every single token. Those stay loaded in RAM, compressed to int4 precision, consuming 9.9 GB. Everything else, the 21,504 routed experts spread across 75 MoE layers, lives on disk across roughly 370 GB of storage. Each expert weighs around 19 MB at int4. When the model’s router selects which experts to activate for a given token, the engine fetches only those specific weight files from disk, runs the computation, then moves on. A per-layer LRU cache keeps recently used experts in memory to reduce redundant reads.

The tradeoff is explicit: speed for accessibility. Streaming weights from an SSD introduces latency that a GPU-resident model wouldn’t face. Colibrì accepts that cost because the alternative — requiring 370-plus gigabytes of RAM — excludes virtually every consumer machine in existence. The disk-streaming approach converts a hardware impossibility into a hardware inconvenience.

The implementation language is pure C with zero external dependencies. No PyTorch, no CUDA runtime, no framework abstraction layer sitting between the inference logic and the metal. That decision carries weight beyond performance. Every line of the engine is auditable without chasing down library source trees. The binary compiles and runs on any machine with a C compiler. A developer reading the code sees exactly what happens when an expert weight is fetched, cached, or evicted — no opacity, no magic. For a project designed to run on hardware that AI infrastructure discourse has written off as inadequate, that transparency is a feature, not just an aesthetic preference.

What mainstream AI coverage keeps getting wrong

Tech journalists reach for parameter count the way financial reporters reach for stock price — as a single number that implies everything. A model with 744 billion parameters sounds like it demands a 744-billion-parameter infrastructure budget. That framing is wrong, and Mixture-of-Experts architecture is why.

In a dense transformer, every parameter activates on every token. In a MoE model like GLM-4, the network routes each token to a small subset of specialized expert layers. GLM-4’s 744B total parameters collapse to roughly 40B active parameters per token in practice. The remaining 704 billion sit idle for any given inference step. Treating the two architectures as equivalent on resource requirements is like comparing a library’s total book count to how many books you read on a Tuesday afternoon.

Most business and policy conversations about AI access carry a buried assumption: frontier-scale language models live in data centers, accessed via API, governed by whoever controls the cloud compute. Colibrì breaks that assumption with a single terminal command. A 744B-parameter model running on a consumer machine with 25GB of RAM, producing coherent output, is a concrete counterexample — not a theoretical one.

The software story is equally misreported. Coverage of AI development treats PyTorch, CUDA, and sprawling ML frameworks as prerequisites, the way coverage of web development once treated enterprise Java as mandatory. Colibrì’s entire inference engine is written in pure C with zero external dependencies. No GPU required. No CUDA stack. No Python runtime. A few thousand lines of C handle quantized weight loading, LRU expert caching, and token streaming. The assumption that local AI inference requires a complex software ecosystem turns out to be a habit of thought, not a technical constraint.

None of this means large language model deployment is trivial. Colibrì runs slowly — disk-streaming 370GB of expert weights across 75 MoE layers is not a latency-optimized path to production. But speed and feasibility are separate questions. The project answers the feasibility question clearly: running a frontier-class open-weight model on consumer hardware, without cloud infrastructure and without a dependency tree, is a solved problem. Mainstream AI coverage has not caught up to that fact.

Why this matters now: the democratisation argument just got a proof of concept

Colibri running a 744-billion-parameter model on a 25 GB consumer machine is not a curiosity — it is a data point that breaks a core vendor assumption. Cloud providers and AI infrastructure companies have spent three years insisting that frontier-scale inference belongs in their racks. A single developer with a C compiler and a fast NVMe drive just demonstrated otherwise.

The 12-to-24-month hardware trajectory makes this more disruptive, not less. Consumer RAM capacities climb, NVMe read speeds improve, and int4 quantisation techniques mature. If the baseline today is running 744B parameters on commodity hardware you can buy at a retail store, the baseline in two years is faster, cheaper, and more accessible. The cloud-centric roadmap vendors are selling to enterprises does not account for that progression.

Local inference at this scale changes the calculus for specific constituencies immediately. Hospitals and legal firms operating under strict data-residency rules cannot send sensitive queries to a third-party API — but they can run a model entirely on-premises without a server room. Developers in regions where dollar-denominated cloud compute is prohibitively expensive gain access to a capability that was, last month, effectively gated behind a subscription. Institutions in countries that restrict outbound data transfer to foreign cloud providers can now run competitive large language model inference without asking permission from anyone.

The MoE architecture point is the one AI labs should sit with. Publishing a Mixture-of-Experts model as an open-weight release is architecturally different from publishing a dense model of equivalent headline parameter count. Because MoE activates only a fraction of parameters per token — roughly 40 billion of GLM-Z1-Rumination-32B’s 744 billion in this case — the memory footprint and disk-streaming approach that Colibri exploits is structurally available to anyone. A dense 744B model cannot be decomposed the same way. Labs releasing MoE architectures open-weight are, whether they intend to or not, releasing models specifically suited to consumer-hardware deployment. That distinction rarely enters the public conversation about responsible open-weight releases, and it should.