[{"content":"Background We ran an experiment to move a persona-deviation detector — a binary classifier that catches when an AI character breaks persona and answers like a generic assistant or refuses a request — that we had been serving on GPU in production onto a Korean NPU. Bottom line up front: after picking apart the causes one by one in the face of the \u0026ldquo;officially unsupported\u0026rdquo; wall, we got the port working. On-device inference clocked in at 9.7ms (fixed 512 tokens, near-zero jitter) — latency and stability that hold up well against our existing cloud-GPU serving. This post is an in-order record of what we tried, where we got stuck, and how we solved it.\nFirst, some context on how we got access to a Korean NPU. Korea supplies domestic AI chips through multiple government-led channels. This isn\u0026rsquo;t marketing copy — there really are several avenues through which a startup or researcher can obtain Rebellions\u0026rsquo; ATOM family, either free or as a pilot.\nK-Cloud Project (MSIT · NIPA, 3 years from 2023) — the \u0026ldquo;AI Semiconductor Farm construction \u0026amp; demonstration\u0026rdquo; program. Three CSPs (Naver Cloud, KT Cloud, NHN Cloud) and three AI-chip makers (Rebellions, Sapeon, FuriosaAI) form a consortium to build and validate domestic-NPU-based high-performance cloud. (ZDNet, NIPA notice) High-Performance Computing Support Program (MSIT · NIPA, ~16.7B KRW in 2026) — provides domestic NPU servers free of charge to selected demand companies. Several supply companies were picked (KT Cloud, Gabia, etc.); Gabia, for instance, provides Rebellions ATOM-Max at no cost. (Etoday, VentureSquare) So through combinations of CSP (KT/NHN/Naver/Gabia…) × program (farm pilot / free provisioning), ATOM, ATOM+, and ATOM-Max are being distributed through a variety of windows. We too obtained 4× ATOM+ cards through one of these paths, in the form of KT Cloud\u0026rsquo;s AI Nexus (Backend.AI-based under the hood).\nWe expected three gains from this port: ① cost savings on always-on GPU serving (government-provided hardware), ② lower, more stable inference latency, and ③ a vendor option beyond NVIDIA GPUs.\nThe problem was what came next.\nThe stack at a glance — from software to hardware Before diving in, let\u0026rsquo;s lay out the layers this work sits on. Turning a single line of text into a verdict passes through several layers running from software (top) to hardware (bottom). (Definitions of each term are collected in the Glossary at the bottom.)\nLayer Role GPU (typical path) NPU (ATOM+, our case) Application Service logic (pooling + classification head) Persona-deviation classifier ← same Model (backbone) Neural net that encodes text into vectors ModernBERT (mmBERT) ← same Integration layer Connects the model definition to the execution layer HuggingFace transformers library (runs directly) optimum-rbln Compiler Converts the model into an executable form torch.compile / TensorRT (optional — runs immediately without it) ahead-of-time compile required via rebel-compiler Runtime Executes the actual computation PyTorch + CUDA rebel.Runtime Hardware Physical accelerator NVIDIA GPU Rebellions ATOM+ (RBLN-CA22) The crux is the compiler layer. On a GPU this stack is mostly transparent and you can run the model as-is, but on an NPU the compiler becomes the gatekeeper. Most of this post is ultimately the story of getting through this one layer.\nA quick aside — what does \u0026ldquo;compile\u0026rdquo; mean here? To keep going with the NPU story, we need to pin down what \u0026ldquo;compiling a model\u0026rdquo; means. In short, it\u0026rsquo;s translating a PyTorch model ahead of time into a form a specific accelerator can execute directly. The GPU default, eager execution (invoking a kernel on the fly for each line of Python), is flexible but has spiky latency; compilation freezes the entire graph into a single deterministic binary, yielding low latency and low jitter. On an NPU this compilation is effectively mandatory, and everything later — having to disable reference_compile or dynamic-shape branches — all stems from this \u0026ldquo;freeze the graph ahead of time\u0026rdquo; constraint.\nThere\u0026rsquo;s a separate post that walks through the concept itself — eager vs. compile, graph capture, op fusion, kernel lowering (OpenAI Triton, etc.), and static shapes — from the ground up with figures and an interactive demo → Neural Net Compilation, Made Simple. Here we only touch on as much as our case requires.\nRecognizing the problem — an NPU must be compiled ahead of time A GPU can run a PyTorch model as-is, but an NPU is different. You have to compile ahead of time for \u0026ldquo;this model, this input size\u0026rdquo; and convert it into an NPU-specific binary. Handling that conversion is Rebellions\u0026rsquo; compiler rebel-compiler, with the integration layer optimum-rbln on top, bridging standard model definitions to it.\nBut our backbone was mmBERT, i.e. the ModernBERT architecture. As it turned out, it was missing from the supported list.\nThe encoders officially supported by optimum-rbln are only bert / distilbert / roberta / xlm-roberta. ModernBERT is not supported. No ModernBERT variant in the Rebellions Model Zoo either. The usual move at this point is to switch to a supported model like XLM-R. But our model\u0026rsquo;s accuracy was already validated on mmBERT, and swapping the backbone carries a heavy re-training and re-validation cost. So we decided to implement support directly.\nHypothesis — \u0026ldquo;is support really impossible?\u0026rdquo; Look into what the \u0026ldquo;unsupported\u0026rdquo; label actually amounts to, and only two kinds of things really block you: (1) there\u0026rsquo;s an op the compiler doesn\u0026rsquo;t understand, or (2) the integration layer lacks a wrapper for that model.\nWe analyzed the elements considered to be ModernBERT\u0026rsquo;s novel additions one by one.\nModernBERT feature Already supported in Alternating local/global attention Gemma2 (1:1), Gemma3 (5:1) Dual RoPE (global θ + local θ) Gemma3 Large vocab (256k) Gemma2/3 (same vocab) GLU/GeGLU MLP LLaMA, Qwen2/3, all Gemma SDPA attention almost every supported decoder Every one of them was an element already present in a supported decoder model (Gemma2/3, etc.). In other words, the compiler\u0026rsquo;s op compatibility is effectively proven, and the real difference is only \u0026ldquo;encoder vs. decoder.\u0026rdquo; And the encoder base class (RBLNModelForMaskedLM) already existed. In the end it was problem (2): only the wrapper was missing.\nOne thing to watch was the attention implementation. ModernBERT defaults to Flash Attention 2 (HuggingFace\u0026rsquo;s flash_attention_2 backend), which is an NVIDIA-CUDA-only kernel that rebel-compiler can\u0026rsquo;t compile for the NPU. Fortunately, forcing PyTorch\u0026rsquo;s standard abstraction API SDPA (scaled_dot_product_attention) lets the compiler recognize it and translate it into its own NPU kernel. It amounts to swapping only the backend, with no accuracy loss.\nVisually, in the computation graph of the mmBERT-base (ModernBERT architecture) we were porting, only two spots were blocking. The rest of the ops (RoPE, GeGLU, etc.) are already supported by the compiler (green).\nflowchart TB IDS[\u0026#34;input_ids · [1,512]\u0026#34;] --\u0026gt; EMB[\u0026#34;Token Embedding + LayerNorm\u0026#34;] MASK[\u0026#34;attention_mask · [1,512]\u0026#34;] -.-\u0026gt; ATTN EMB --\u0026gt; LN1 subgraph LYR[\u0026#34;Encoder layer × 22 (mmBERT-base = ModernBERT)\u0026#34;] direction TB LN1[\u0026#34;LayerNorm (pre-norm)\u0026#34;] --\u0026gt; ROPE[\u0026#34;RoPE (global θ / local θ)\u0026#34;] ROPE --\u0026gt; ATTN[\u0026#34;Attention\u0026lt;br/\u0026gt;local (sliding window) · global (every 3rd layer)\u0026#34;] ATTN --\u0026gt; RES1[\u0026#34;＋ residual\u0026#34;] RES1 --\u0026gt; LN2[\u0026#34;LayerNorm\u0026#34;] --\u0026gt; MLP[\u0026#34;GeGLU MLP\u0026#34;] --\u0026gt; RES2[\u0026#34;＋ residual\u0026#34;] end RES2 --\u0026gt; FLN[\u0026#34;final LayerNorm\u0026#34;] --\u0026gt; POOL[\u0026#34;mean pooling\u0026#34;] --\u0026gt; HEAD[\u0026#34;linear head → logits(2)\u0026#34;] ATTN -.- NOTE1[\u0026#34;⚠ Problem ① Flash Attention 2 (CUDA-only)\u0026lt;br/\u0026gt;→ replace with SDPA · disable sliding_window_mask·unpadding (dynamic shape)\u0026#34;] LYR -.- NOTE2[\u0026#34;⚠ Problem ② reference_compile=False\u0026lt;br/\u0026gt;(remove torch.compile inside forward)\u0026#34;] class ATTN problem class ROPE,MLP okc class NOTE1,NOTE2 note classDef problem fill:#7f1d1d,stroke:#f87171,stroke-width:3px,color:#fff classDef okc fill:#14532d,stroke:#4ade80,color:#fff classDef note fill:#3a2f0b,stroke:#e0a020,color:#fbe8a6In other words, it wasn\u0026rsquo;t the whole architecture that was the problem — only two spots collided with NPU compilation: the attention kernel (FA2) and the torch.compile inside forward (reference_compile). Stripping these two out in the adapter was the next step.\nWriting the adapter — three things to control We wrote the ModernBERT adapter by referencing the pattern of the sibling class RBLNRobertaForMaskedLM. There are three points to control.\nDisable incompatible options at the config level — attn_implementation=\u0026quot;sdpa\u0026quot;, reference_compile=False (ModernBERT calls torch.compile inside forward, which breaks trace/export). Block incompatible kwargs at the wrapper level — explicitly disable unpadding/sequence-packing args (indices, cu_seqlens, sliding_window_mask, etc.) to remove dynamic-shape branches. Simplify the input signature — just input_ids + attention_mask. Up to here you can verify without an NPU. Extracting the graph on a MacBook with torch.jit.trace and torch.export, the error vs. eager across two different inputs was max|diff| = 0.0 — a numerically identical graph. Once you\u0026rsquo;ve confirmed this much, you\u0026rsquo;re confident the port is feasible. All that\u0026rsquo;s left is real-hardware validation.\nOn-hardware compile — the first try failed We attached to an ATOM+ session (the container ships rebel-compiler and optimum-rbln pre-installed, so no separate install was needed) and ran the compile. The first attempt failed with this error:\nValueError: Configuration for RBLNModernBertForMaskedLMConfig not found. Tracing the cause: the optimum-rbln we had referenced was the latest main, but what was actually installed in the container was the 0.10.2 release, and that version resolved config classes only via a name-based registry (a top-level attr on optimum.rbln, or the internal CONFIG_MAPPING). Our custom adapter\u0026rsquo;s config wasn\u0026rsquo;t registered there, so it failed as \u0026ldquo;not found.\u0026rdquo; A trap created by SDK version drift.\nThe fix was one line — pin the config class directly on the model class so it skips the registry lookup:\nclass RBLNModernBertForMaskedLM(RBLNModelForMaskedLM): _rbln_config_class = RBLNModernBertForMaskedLMConfig # bypass the registry Rerun, and this time it passed:\n[rebel-compiler] Target NPU: RBLN-CA22 [rebel-compiler] Tensor parallel size: 1 Computation graph generation ████████████ 100% Computation graph optimization ████████████ 100% saved -\u0026gt; compiled_model.rbln The physical device confirmed via rbln-smi was RBLN-CA22 (ATOM+), 15.7GiB. The moment an \u0026ldquo;unsupported\u0026rdquo; model compiled on a Korean NPU. The compile itself finished in a dozen-odd seconds.\nApplying it for real — porting the whole classifier So far we had only compiled the backbone (MaskedLM). What we actually serve is a persona-deviation classifier with mean pooling + a linear head layered on top of it. Since that\u0026rsquo;s a custom nn.Module rather than a standard HuggingFace architecture, it was cleaner to use the compiler\u0026rsquo;s low-level API directly instead of optimum-rbln\u0026rsquo;s high-level path.\nflowchart TD T[\u0026#34;Raw text\u0026#34;] --\u0026gt; TOK[\u0026#34;Tokenizer\u0026lt;br/\u0026gt;input_ids · attention_mask · [1, 512]\u0026#34;] TOK --\u0026gt; BB subgraph G[\u0026#34;Compile unit — rebel.compile_from_torch ⇒ compiled_model.rbln (runs on ATOM+ NPU)\u0026#34;] direction TB BB[\u0026#34;mmBERT backbone · SDPA\u0026#34;] --\u0026gt; MP[\u0026#34;mean pooling\u0026#34;] MP --\u0026gt; LH[\u0026#34;linear head\u0026#34;] end LH --\u0026gt; LOG[\u0026#34;logits (2)\u0026#34;] LOG --\u0026gt; P[\u0026#34;softmax → deviation probability\u0026#34;]The tokenizer runs on CPU, and below it the backbone + pooling + head are bundled into a single compile unit that runs on the NPU. A single call to rebel.compile_from_torch(model, input_info=[(\u0026quot;input_ids\u0026quot;,[1,512],\u0026quot;int64\u0026quot;), (\u0026quot;attention_mask\u0026quot;,[1,512],\u0026quot;int64\u0026quot;)]) compiled these three into a single graph.\ncompiled_model.rbln is a serialized binary, so you can\u0026rsquo;t just open it up, but if you unfold the graph the compiler handles internally, it looks roughly like this (conceptual representation).\n# compiled_model.rbln — internal graph (conceptual representation) target: RBLN-CA22 (ATOM+) tensor_parallel: 1 dtype: fp16 in %input_ids : i64[1, 512] in %attention_mask : i64[1, 512] %emb = rbln.embedding %input_ids -\u0026gt; f16[1,512,768] %h0 = rbln.layernorm %emb # × N encoder layers (attention → mlp): %a = rbln.sdpa_attention %h{i}, mask=%attention_mask @kernel=attn_fused_ca22 %m = rbln.geglu_mlp %a @kernel=geglu_fused_ca22 ... %enc = rbln.layernorm %hN -\u0026gt; f16[1,512,768] %pool = rbln.mean_pool %enc, mask=%attention_mask -\u0026gt; f16[1,768] %log = rbln.linear %pool, W=const[768,2] -\u0026gt; f16[1,2] out %log The compile steps described earlier show up plainly here — the ops were lowered into ATOM+-specific kernels of the form @kernel=..._ca22, and every tensor shape is statically fixed to [1,512,...]. And the most important check — do the CPU and NPU predictions agree?\nInput type CPU deviation prob NPU deviation prob Deviation utterance 1 1.0000 1.0000 Normal utterance 1 0.0007 0.0007 Deviation utterance 2 (Korean) 1.0000 1.0000 Normal utterance 2 (Korean) 0.0002 0.0002 The numerical error at the logits level was at most ~4e-2 (a characteristic of NPU FP16 arithmetic), but the probabilities and decisions matched to four decimal places. The predictions reproduced exactly.\nBenchmark Same model, batch=1, fixed 512 tokens.\nMetric ATOM+ (on-device) Mean latency 9.67 ms p50 / p99 9.66 / 9.73 ms max 10.6 ms Single-stream throughput 103.5 inf/s As impressive as the numbers was the stability of the distribution. p50 9.66ms / p99 9.73ms / max 10.6ms — essentially no jitter. Because it runs a fixed compiled graph deterministically, the latency doesn\u0026rsquo;t spike and stays flat. It was comfortably competitive even against the cloud-GPU serving we had been using. Power draw is also around 85W per card.\nDid the expectations hold up? To sum up, of the three, low latency and stability were confirmed beyond expectations (9.67ms, near-zero jitter). The vendor option opened up too, demonstrating that \u0026ldquo;even an unsupported model can be ported.\u0026rdquo; The serving configuration for cost savings is currently in preparation.\nLessons \u0026ldquo;Unsupported\u0026rdquo; usually means \u0026ldquo;no wrapper.\u0026rdquo; If the model is built from a combination of ops the compiler understands, you can port it with a thin adapter. The power of SDPA standardization. Drop down to PyTorch\u0026rsquo;s standard abstraction API instead of a vendor-locked kernel (FA2), and the backend (GPU or NPU) translates it into its own kernel. This is the crux of portability. Verify as much as possible locally, then go to hardware. Confirming graph equivalence (zero error) via trace/export locally drastically shrinks the surface you have to debug on real hardware. Watch out for SDK version drift. The internal API of the docs/latest-main and the actually-deployed release can differ. This time, a single line about the config-registry approach blocked the first compile. There are probably plenty of teams leaving a Korean NPU in a \u0026ldquo;received but can\u0026rsquo;t use it\u0026rdquo; state. It wasn\u0026rsquo;t without trial and error, but at least for encoder-family models it can definitely be ported. Now that the government has laid out the hardware through so many channels, it\u0026rsquo;s worth a serious attempt.\nGlossary NPU (Neural Processing Unit) — a chip specialized for neural-network computation. Unlike a general-purpose GPU, you fix the model to run and the input size ahead of time, compile it, and run it as a dedicated binary. Encoder model / ModernBERT / mmBERT — a neural net that converts text into a vector representation (understanding-centric, not generation). ModernBERT is the latest BERT-family architecture, and mmBERT is its multilingual variant. Compile (in the NPU context) — the process of converting a PyTorch model into an NPU-specific execution binary (.rbln) fitted to a specific input shape. Contrasts with a GPU\u0026rsquo;s immediate execution. rebel-compiler / optimum-rbln / rebel.Runtime — respectively, Rebellions\u0026rsquo; compiler, the integration layer that connects a HuggingFace model to the compiler, and the runtime that executes the compiled binary on the NPU. SDPA vs. Flash Attention 2 — both are attention implementations. The FA2 here means HuggingFace\u0026rsquo;s flash_attention_2 backend (the NVIDIA-CUDA-only flash-attn kernels), which rebel-compiler can\u0026rsquo;t compile. SDPA (scaled_dot_product_attention) is a PyTorch standard abstraction API that each backend can translate into its own kernel, so it\u0026rsquo;s highly portable. (Note: RBLN also has its own flash_attn option for decoder KV-cache attention, but that\u0026rsquo;s different from HF\u0026rsquo;s FA2 backend and doesn\u0026rsquo;t apply to encoders.) RoPE / GLU·GeGLU — respectively, an embedding technique that injects positional information via rotation, and an MLP structure with gating applied. Both are widely used in modern transformers and are already supported by the Rebellions compiler. MaskedLM — the base form of an encoder (backbone) trained to mask part of a sentence and restore it. You put a pooling/classification head on top to make it a classifier. mean pooling — a method that averages per-token vectors to summarize a sentence into a single vector. ATOM+ / RBLN-CA22 — Rebellions\u0026rsquo; NPU card (ATOM+) and its internal target identifier (RBLN-CA22). References K-Cloud Project / AI Semiconductor Farm construction \u0026amp; demonstration — ZDNet, NIPA integrated notice 2026 High-Performance Computing Support Program (free domestic-NPU provisioning) — Etoday, VentureSquare Rebellions optimum-rbln (the integration layer the adapter is based on) — GitHub Rebellions Model Zoo — GitHub Rebellions RBLN SDK docs — docs.rbln.ai ","permalink":"http://blog.caveduck.io/posts/modernbert-on-rebellions-npu/","summary":"Moving a persona-deviation detector (an mmBERT-based classifier) that we served on GPU in production onto a Korean NPU (Rebellions ATOM+) — the journey from recognizing the problem to writing the adapter, compiling on real hardware, and benchmarking.","title":"That Free Korean NPU the Government Hands Out — Can You Actually Use It? Notes From Porting an Unsupported Model"},{"content":" This post is a concept explainer that branched off from Porting an Unsupported Model to a Korean NPU. Where that post is about \u0026ldquo;how we actually broke through,\u0026rdquo; this one unpacks from scratch the \u0026ldquo;what does it even mean to compile a model\u0026rdquo; that underlies it. It\u0026rsquo;s written to read smoothly for anyone with a bit of deep learning background.\nStarting from the Familiar Word \u0026ldquo;Compile\u0026rdquo; In programming, compilation is the process of translating ahead of time the human-readable source code (C, Rust, etc.) into machine code that the CPU executes directly. You do the translation up front, and in return, execution is fast.\nNeural network compilation is exactly the same idea. You take a model written in PyTorch and translate it ahead of time into a form that a specific accelerator (GPU, NPU, etc.) can execute directly. The one thing that differs is the starting point of the translation.\nInput to a regular compiler = text source code Input to a neural network compiler = computation graph A computation graph is a directed graph of chained tensor operations, like \u0026ldquo;matrix-multiply this tensor → then attention → then normalization → …\u0026rdquo;. Think of it as the model\u0026rsquo;s forward() unfolded into a picture.\nThat said, \u0026ldquo;a pipeline flowing in a straight line\u0026rdquo; is a story about the simple models in textbooks; a real model is a graph where multiple modules branch and merge. For example, an image editing model (of the Qwen-Image-Edit family) sends the input image, edit prompt, and timestep through their own encoders, merges them into a single backbone, and loops through denoising with feedback.\nflowchart TD IMG[\u0026#34;Input image\u0026#34;] --\u0026gt; VAE[\u0026#34;VAE encoder\u0026#34;] IMG --\u0026gt; VIT[\u0026#34;Vision encoder\u0026lt;br/\u0026gt;Qwen2.5-VL\u0026#34;] TXT[\u0026#34;Edit prompt\u0026#34;] --\u0026gt; TENC[\u0026#34;Text encoder\u0026lt;br/\u0026gt;LLM\u0026#34;] TS[\u0026#34;timestep t\u0026#34;] --\u0026gt; TEMB[\u0026#34;Time embedding\u0026#34;] VAE --\u0026gt; LAT[\u0026#34;Image latent tokens\u0026#34;] VIT --\u0026gt; COND[\u0026#34;Semantic condition tokens\u0026#34;] TENC --\u0026gt; TTOK[\u0026#34;Text tokens\u0026#34;] LAT --\u0026gt; BB COND --\u0026gt; BB TTOK --\u0026gt; BB TEMB --\u0026gt; BB subgraph BB[\u0026#34;MMDiT backbone · multimodal joint attention × N blocks\u0026#34;] direction TB JA[\u0026#34;Joint self-attention\u0026lt;br/\u0026gt;image ↔ text\u0026#34;] --\u0026gt; FF[\u0026#34;MLP\u0026#34;] FF -. residual .-\u0026gt; JA end BB --\u0026gt; EPS[\u0026#34;Predicted noise ε\u0026#34;] EPS --\u0026gt; LOOP{\u0026#34;Denoising loop\u0026#34;} LOOP --\u0026gt;|\u0026#34;next step t−1\u0026#34;| BB LOOP --\u0026gt;|\u0026#34;done\u0026#34;| VDEC[\u0026#34;VAE decoder\u0026#34;] VDEC --\u0026gt; OUT[\u0026#34;Edited image\u0026#34;]The same input (the image) splits into two branches feeding different encoders, merges at the backbone together with the text and timestep, and inside the blocks there are residual connections while outside there\u0026rsquo;s a denoising feedback loop. The graph a compiler actually faces is this kind of tangle of branching, merging, and repetition.\nHere\u0026rsquo;s one important fact. The front end of a regular compiler — lexical analysis (splitting source text into tokens) and parsing (turning tokens into a syntax tree) — does not exist in neural network compilation. That\u0026rsquo;s because the input is already given as a structured graph (equivalent to the AST/IR in a regular compiler). So neural network compilation skips the entire front end of the compiler pipeline and effectively starts from the IR optimization and code generation stages.\nSo Why Compile at All? — The Cost of Eager Execution PyTorch\u0026rsquo;s default execution mode is eager execution. Each time a line of Python code runs, it immediately invokes the corresponding operator kernel on the spot. It\u0026rsquo;s closer to an interpreter that translates one sentence at a time as you converse — flexible, and easy to debug by printing intermediate values along the way.\nThe problem is cost. For every single operation, Python interpretation → kernel dispatch → execution repeats, and this overhead adds up a little differently each time (i.e., latency spikes, jitter). General-purpose GPUs have kernels that are so fast and flexible that this approach runs well enough, but specialized accelerators like NPUs deliver their real performance only when they know in advance what to run and in what order.\nCompilation is the act of turning this interpreter into a translated book. If you receive the whole graph before execution and freeze it into a single deterministic execution unit, there\u0026rsquo;s nothing to decide on the fly for each request, so execution doesn\u0026rsquo;t spike.\nSeeing it is faster than talking about it. Below, run the same model many times in both eager and compiled modes and compare how the latency distributions differ.\nEager dispatches ops one at a time, so its average is higher and its distribution is spread out widely. Compilation executes a single graph deterministically, so it converges to nearly a single point. This is where the value of compilation shows up in real services that need low latency and low jitter.\nWhat the Compiler Does in Between When you hand over a single model, the compiler goes through roughly five stages.\nflowchart TD A[\u0026#34;① Graph capture\u0026lt;br/\u0026gt;trace / export\u0026#34;] --\u0026gt; B[\u0026#34;② Operator fusion \u0026amp; optimization\u0026#34;] B --\u0026gt; C[\u0026#34;③ Lowering to hardware kernels\u0026lt;br/\u0026gt;lowering\u0026#34;] C --\u0026gt; D[\u0026#34;④ Fix static shapes\u0026#34;] D --\u0026gt; E[\u0026#34;⑤ Memory planning\u0026#34;] E --\u0026gt; F([\u0026#34;Execution binary\u0026#34;])① Graph Capture — Turning Python into a Graph First, you extract a graph of what operations the model actually performs. There are two approaches.\ntrace (torch.jit.trace): Actually run an example input through once and record the operations it passed through. Simple, but for things like if branches, only \u0026ldquo;the path taken that time\u0026rdquo; is captured. export (torch.export): Statically analyze the code to extract a graph that includes control flow. More robust. ② Operator Fusion — Combining Multiple Operations into One Adjacent operations are merged into one. For example, running matrix-multiply → add bias → activation function separately writes and reads the intermediate result to memory three times. Merging these into a single fused kernel reduces the memory round trips and makes it much faster.\nflowchart LR subgraph before[\u0026#34;Before fusion — 3 kernels, many memory round trips\u0026#34;] direction LR M1[\u0026#34;matmul\u0026#34;] --\u0026gt; B1[\u0026#34;+bias\u0026#34;] --\u0026gt; A1[\u0026#34;gelu\u0026#34;] end subgraph after[\u0026#34;After fusion — 1 kernel\u0026#34;] direction LR F1[\u0026#34;fused_matmul_bias_gelu\u0026#34;] end before -.optimize.-\u0026gt; after③ Lowering to Hardware Kernels Each standard operation (matrix multiply, attention…) is translated into instructions and kernels specific to that chip. A GPU example gives you the feel of it. The default backend of PyTorch\u0026rsquo;s torch.compile (TorchInductor) lowers operations to OpenAI Triton. Triton is a language for describing GPU kernels with Python-like syntax; the compiler generates a kernel matching the fused operations as Triton code on the spot, then lowers that further into a GPU binary (PTX/cubin). Instead of calling a vendor\u0026rsquo;s prebuilt library kernel (cuBLAS, etc.), it generates a kernel tailored to the graph. NPU compilers follow the same idea — the only difference is that the code generation backend is specific to that chip rather than Triton.\n④ Fix Static Shapes You nail down the input sizes in advance (e.g., [1, 512]). Once the sizes are fixed, branches at runtime disappear and a deterministic path is created, and fusion and memory planning can be done more aggressively too. This is why \u0026ldquo;fixed 512 tokens\u0026rdquo; keeps showing up in the NPU-related posts. The cost is flexibility — inputs that go beyond the size decided at compile time can\u0026rsquo;t be fed in (unless you recompile or pad).\n⑤ Memory Planning The placement and reuse of the buffers each operation will use are finalized before execution. Since memory isn\u0026rsquo;t allocated and freed on the fly during execution, it\u0026rsquo;s that much faster and more consistent.\nWhat Does the Compiled Artifact Look Like The result is a binary the hardware consumes directly (for Rebellions, a .rbln). It\u0026rsquo;s serialized, so you can\u0026rsquo;t just open it and read it, but if you conceptually unfold the internal graph, it looks like this.\n# compiled_model — internal graph (conceptual representation) target: RBLN-CA22 dtype: fp16 in %input_ids : i64[1, 512] in %attention_mask : i64[1, 512] %emb = embedding %input_ids -\u0026gt; f16[1,512,768] %h0 = layernorm %emb # × N encoder layers: %a = sdpa_attention %h{i}, mask=%attention_mask @kernel=attn_fused %m = geglu_mlp %a @kernel=geglu_fused ... %pool = mean_pool %enc, mask=%attention_mask -\u0026gt; f16[1,768] %log = linear %pool, W=const[768,2] -\u0026gt; f16[1,2] out %log The stages we saw earlier show up directly — operations are mapped to dedicated kernels in the @kernel=..._fused form, and every tensor shape is statically fixed at [1,512,...].\nWhat Already Exists — ONNX and the Compilation Ecosystem So far we\u0026rsquo;ve lumped \u0026ldquo;the compiler\u0026rdquo; into one thing, but in reality several standards and tools form layers. The one that frequently appears at the hub is ONNX (Open Neural Network Exchange).\nONNX is not a compiler but an open standard format for holding a computation graph. No matter where a model is built — PyTorch, TensorFlow, wherever — once you export it to ONNX, various runtimes and compilers that can read that graph execute it with their own backends. It\u0026rsquo;s a common language that decouples the framework a model was built in from where it gets deployed.\nflowchart TD PT[\u0026#34;PyTorch\u0026#34;] --\u0026gt; ONNX TF[\u0026#34;TensorFlow\u0026#34;] --\u0026gt; ONNX JAX[\u0026#34;JAX\u0026#34;] --\u0026gt; ONNX ONNX[\u0026#34;ONNX · common graph standard\u0026#34;] --\u0026gt; ORT[\u0026#34;ONNX Runtime\u0026#34;] ONNX --\u0026gt; TRT[\u0026#34;TensorRT · NVIDIA\u0026#34;] ONNX --\u0026gt; OV[\u0026#34;OpenVINO · Intel\u0026#34;] ONNX --\u0026gt; NPU[\u0026#34;Vendor NPU compiler\u0026#34;]Here are the tools you\u0026rsquo;ll often run into, organized by their nature.\nTool Nature Role ONNX Standard format Graph exchange across frameworks (not a compiler itself) ONNX Runtime Runtime + optimization Optimizes the graph, then runs it on various execution backends (Execution Providers) TensorRT Compiler + runtime High-performance inference exclusive to NVIDIA GPUs Apache TVM Compiler stack Open compiler targeting diverse backends (CPU/GPU/accelerators) XLA Compiler JAX/TensorFlow (and torch/XLA), TPU-centric torch.compile Compiler Built into PyTorch — TorchInductor → Triton OpenVINO Compiler + runtime Targets Intel CPU/iGPU/NPU optimum-rbln / rebel-compiler Vendor stack Targets Rebellions NPU (the case study of this series) The big picture is usually a flow of ① extracting a graph from a framework (capture/export) → ② passing through a common representation like ONNX, or going straight → ③ compiling for the target backend. Most NPU vendors also accept ONNX input. In this series, however, instead of going through the extra ONNX layer, we chose the path of handing the PyTorch-native graph (torch.export) directly to the vendor compiler — when dealing with a model whose architecture is subtly misaligned, cutting one step of format conversion made the debugging surface area smaller.\nTrade-offs — What You Gain and What You Lose eager (interpreter) compiled (translated book) Latency High, large jitter Low, consistent Flexibility Arbitrary Python \u0026amp; dynamic shapes OK Fixed graph \u0026amp; fixed shapes Debugging Easy (inspect values instantly) Hard (the graph is frozen) Setup cost None Compilation time required In short, compilation is turning a \u0026ldquo;flexible interpreter\u0026rdquo; into a \u0026ldquo;deterministic execution binary.\u0026rdquo; You give up flexibility and gain low latency, low jitter, and efficiency.\nSo When Do You Compile Research and prototyping, when input sizes change every time → eager is enough. Flexibility is valuable. Real-service low-latency inference, edge and on-device, specialized accelerators (NPUs) → compilation wins. For NPUs in particular, compilation is effectively mandatory. In other words, \u0026ldquo;compilation is always the answer\u0026rdquo; is not true. When flexibility matters, eager is the fit; when latency, stability, and efficiency matter, compilation is. How we navigated this trade-off while actually running a real model on a Korean NPU continues in the porting experiment writeup.\n","permalink":"http://blog.caveduck.io/posts/neural-net-compilation/","summary":"What exactly does it mean to \u0026lsquo;compile a model\u0026rsquo;? From the difference with eager execution to graph capture, operator fusion, kernel lowering, static shapes, and memory planning — we walk through it step by step with diagrams and an interactive demo.","title":"Neural Network Compilation, Made Simple — How a PyTorch Model Becomes Hardware Language"},{"content":" This is the third post (#3) in the Efficient ML series. If data types → quantization were about \u0026ldquo;representing numbers with fewer bits,\u0026rdquo; this time we go further and cut the connections themselves — pruning. The code lives in the pruning.ipynb notebook in github.com/warpspaceinc/efficient-ml-practice, and every plot below is our own numbers, measured by running that code on Colab (A100) ourselves.\nThe \u0026ldquo;brain damage\u0026rdquo; of 1989 When you talk about pruning, there\u0026rsquo;s one paper you can\u0026rsquo;t skip. Its title is downright gruesome — Optimal Brain Damage. The authors are Yann LeCun, John Denker, and Sara Solla of AT\u0026amp;T Bell Labs, and it was presented at NeurIPS 1989.\nStop and think about it for a second and it\u0026rsquo;s astonishing. 1989. That\u0026rsquo;s 36 years ago. Before the World Wide Web was released to the world, before LeCun became famous for CNNs (LeNet), back when the term \u0026ldquo;deep learning\u0026rdquo; didn\u0026rsquo;t even exist. That same LeCun is now Meta\u0026rsquo;s Chief AI Scientist and a Turing Award winner. The core idea of pruning — \u0026ldquo;if you cut away the less important connections in a neural network, you get a smaller, better-generalizing network\u0026rdquo; — had already been formalized that long ago.\nThe paper\u0026rsquo;s premise also touches on a biological metaphor. The human brain doesn\u0026rsquo;t just keep piling on synapses either. The number of synapses per neuron explodes from about 2,500 in infancy → up to 15,000 around age 2–4 → and then falls back to about 7,000 in adulthood. This \u0026ldquo;pruning\u0026rdquo; — snipping away the useless connections — is what actually produces an efficient brain.\nThis article follows how that 36-year-old insight becomes reality today, as 2:4 sparsity on NVIDIA GPUs, through three questions.\nWhat to cut (criterion) How much you can cut (retraining and the surprise of compression) How to cut so the hardware actually speeds up (granularity and 2:4) What pruning is \u0026amp; the 5-step framework Pruning means deleting a connection by setting its weight to 0. You take a dense network and make it sparse, saving on storage, computation, and power. Formally, it\u0026rsquo;s the problem of \u0026ldquo;minimizing the loss while keeping the number of nonzero weights at or below $N$.\u0026rdquo;\n$$\\arg\\min_{W_P} L(x; W_P) \\quad \\text{s.t.} \\quad \\lVert W_P\\rVert_0 \u003c N$$Since Han et al. (NeurIPS 2015), pruning is usually organized into 5 steps: ① train → ② granularity (how) → ③ criterion (what) → ④ ratio (how much) → ⑤ fine-tune (recovery). This article walks through the core of it in the order criterion → ratio/recovery → granularity.\nThe motivation is the same as in the quantization article. Weights are mostly clustered near 0, and memory access is hundreds of times more expensive than computation (Horowitz: DRAM 640pJ vs int ADD 0.1pJ). So it\u0026rsquo;s natural to ask: \u0026ldquo;so many weights are almost 0 — do we really need all of them?\u0026rdquo;\n1. What to cut — Criterion The first question to answer is \u0026ldquo;which weights do we erase?\u0026rdquo; There are three candidates.\nMagnitude: importance $= |w|$. If it\u0026rsquo;s small, erase it. Cheap and simple. OBD saliency (second-order / Hessian): this is OBD\u0026rsquo;s answer. It uses a Taylor expansion to approximate how much the loss would grow if you erased each weight. Random: at random. A lower bound for comparison. OBD\u0026rsquo;s derivation goes like this. Expand the loss $L$ to second order in the weight change $\\delta w$:\n$$\\delta L = \\sum_i g_i\\,\\delta w_i + \\frac{1}{2}\\sum_i h_{ii}\\,\\delta w_i^2 + \\frac{1}{2}\\sum_{i\\neq j} h_{ij}\\,\\delta w_i \\delta w_j + O(\\lVert\\delta w\\rVert^3)$$Here we make three assumptions — convergence complete (the first-order term $g_i \\approx 0$), diagonal approximation (ignore the cross terms $h_{ij}$), and second-order approximation (ignore everything third-order and up). Then the loss increase from setting a single weight to 0 ($\\delta w_i = -w_i$) — the saliency — comes out cleanly.\n$$\\text{saliency}_{w_i} = \\frac{1}{2}\\, h_{ii}\\, w_i^2$$It says the larger the weight ($w_i^2$), and the steeper the loss curvature ($h_{ii}$), the more important it is. The catch is that the Hessian $h_{ii}$ is expensive — so in practice we approximate the diagonal Hessian with the empirical Fisher (the mean of squared gradients). Our experiment did the same.\nMeasured directly: comparing the three criteria After training an MNIST MLP (784→512→512→10), we raised the sparsity under each of the three criteria and measured accuracy (without retraining).\nThe key things to read:\nRandom collapses the moment it passes 50%. Proof that what you cut is decisive. Magnitude and OBD nearly overlap. Both hold their accuracy up to 90%. At the extreme (98%) OBD\u0026rsquo;s second-order insight edges ahead (44.6% vs 29.3%), but right up until that point magnitude is actually marginally better. Try it yourself below. As you raise sparsity with the slider, cut synapses disappear from the network on the right, and neurons left with no connection are removed. The compute (MACs·FLOPs) drops along with them — but that saving is only real if the hardware actually skips the zeros. Switch the criterion between magnitude ↔ random and, at the same sparsity (same MAC count), you see which connections and neurons vanish first change.\nThis result is exactly the answer to \u0026ldquo;why does practice use magnitude instead of OBD.\u0026rdquo; OBD is theoretically elegant, but the Hessian is expensive. Magnitude, on the other hand, is free and this powerful. That\u0026rsquo;s why, 36 years later, most pruning is still magnitude-based. The door OBD opened (\u0026ldquo;let\u0026rsquo;s principled-ly choose what to cut\u0026rdquo;) ended up being walked through, in practice, by the simplest criterion of all.\n2. How much can you cut — retraining and the surprise of compression Cut a lot at once and accuracy naturally drops. But if you repeat cut → retrain (fine-tune), the story changes completely. The retraining learning rate is usually set low, at about 1/10 to 1/100 of the original.\nprune only (gray): cut without retraining and at 95% you fall to 88%. iterative prune+retrain (green): cut a little at a time and retrain each round, and even cutting away 95% of the weights (= 20× fewer parameters) holds essentially the same accuracy as dense (97.3% vs 97.6%). Keeping only one-twentieth of the weights and losing no performance is precisely why pruning is still researched today. Look at the weight distribution after pruning and you can see what happened at a glance — a giant spike appears exactly at 0.\nThe work that pushed this \u0026ldquo;cut and re-learn\u0026rdquo; idea to the extreme is Deep Compression (Han et al., 2016). Iterative pruning raised AlexNet\u0026rsquo;s pruning limit from 5× to 9×, and adding quantization and Huffman coding on top achieved up to dozens of times.\n3. How to cut so it speeds up — Granularity and hardware There\u0026rsquo;s a trap here that absolutely must be pointed out. Getting sparsity from pruning does not automatically make things faster.\nThink about it carefully: \u0026ldquo;cutting\u0026rdquo; a weight is really no different from putting a 0 in its place. And if you just feed a 0 into a multiply-accumulate unit — $0 \\times a = 0$ — the number of operations doesn\u0026rsquo;t drop at all. The accuracy experiments above actually set the weights to 0 and then ran them through the dense kernel as-is, so accuracy was measured but neither storage nor speed shrank one bit (exactly the \u0026ldquo;fake quantization\u0026rdquo; situation from the quantization article). Whether you multiply by 0 or by 0.3, from the multiplier\u0026rsquo;s point of view it\u0026rsquo;s the same single multiplication.\nTo reap the benefit, you have to not store the 0, not multiply it, and skip it entirely. But that turns out to be hard. Pruning so far has been the unstructured kind, where zeros scatter anywhere, so the compression ratio is best — but from the GPU\u0026rsquo;s point of view, the positions of the zeros are irregular, so you can\u0026rsquo;t skip them. The cost of finding \u0026ldquo;where is the next zero\u0026rdquo; every time is more expensive than the single multiplication you\u0026rsquo;d save, so in the end running dense is better.\nThat\u0026rsquo;s why this principle always tags along with pruning:\nPruning only yields a real payoff when the hardware supports that sparsity pattern. You have to know what the hardware you\u0026rsquo;ll use accelerates, and cut accordingly.\nDedicated hardware for sparse computation (EIE, ISCA 2016, etc.) has been researched over the years, and the most widely used answer today is NVIDIA\u0026rsquo;s 2:4 structured sparsity.\n2:4 pruning The rule is simple. For every 4 consecutive weights, set exactly 2 to 0 (50% sparsity). Because the positions of the zeros are regular — \u0026ldquo;2 out of every 4\u0026rdquo; — NVIDIA\u0026rsquo;s Sparse Tensor Core (from Ampere / A100 onward) accelerates this pattern in hardware. Storage is efficient too — you only need half the nonzeros plus a 2-bit index pointing to each value\u0026rsquo;s position.\nHere\u0026rsquo;s how to build a mask that cuts the smaller 2 of every 4 columns by magnitude.\ndef two_four_mask(weight): \u0026#34;\u0026#34;\u0026#34;A 2:4 mask that keeps only the 2 largest |w| per group of 4 columns.\u0026#34;\u0026#34;\u0026#34; w = weight.detach().abs(); R, C = w.shape mask = torch.zeros_like(weight) for j in range(0, C - C % 4, 4): idx = w[:, j:j+4].topk(2, dim=1).indices # the larger 2 of each group of 4 mask[:, j:j+4].scatter_(1, idx, 1.0) return mask The result of applying 2:4 to our MLP: dense 97.6% → 2:4 (no retraining) 96.8% → 2:4 + retraining 98.2%. Even cutting half regularly, retraining fully recovers it.\nUsing 2:4 from Python PyTorch provides an API to store the 2:4 pattern compressed and multiply it on the Sparse Tensor Core (requires fp16, Ampere+).\nfrom torch.sparse import to_sparse_semi_structured mask = torch.Tensor([0, 0, 1, 1]).tile((64, 16)).bool() W = (torch.rand(64, 64) * mask).half().cuda() Ws = to_sparse_semi_structured(W) # compressed to nonzeros + 2-bit indices x = torch.rand(64, 64).half().cuda() torch.mm(Ws, x) # uses the Sparse Tensor Core Does it really speed up — benchmarked directly On an A100 we measured dense vs 2:4 sparse matmul speed by size. The result is honest and instructive.\nFor small matrices (N \u0026lt; 4096), 2:4 is actually slower. The compression and indexing overhead outweighs the benefit. The benefit only shows up on large matrices. At N=8192 it\u0026rsquo;s about 1.72× faster. This is the concrete meaning of \u0026ldquo;you have to know your hardware and workload.\u0026rdquo; The very same 2:4 pruning only speeds up on an Ampere-or-later GPU, and only for sufficiently large matrices. On Turing (T4) or small matrices there\u0026rsquo;s no benefit — or even a loss. Pruning isn\u0026rsquo;t purely an algorithm problem; it\u0026rsquo;s a problem you have to design together with the hardware from start to finish.\n4. Pruning × quantization are orthogonal — Deep Compression We\u0026rsquo;ve looked at pruning up to here, and in the earlier quantization article we reduced the bits. The key point is that these two are orthogonal. Pruning decides \u0026ldquo;which weights to erase,\u0026rdquo; while quantization decides \u0026ldquo;how many bits to represent the surviving weights with.\u0026rdquo; The axes are different, so using them together multiplies the gains.\nThe best demonstration of this is the 3-stage pipeline of Deep Compression (Han et al., 2016).\nPruning + retraining — erase the weak connections and re-learn to recover. Quantization + retraining — group the surviving weights into a handful of K-Means codebook entries (retraining the codebook) and store only the indices. Huffman coding — since the index distribution is skewed to one side, compress once more, losslessly. We ran it on our own MLP. Watching how the fc2 weight distribution changes at each stage, you can see at a glance that pruning and quantization touch different parts of the distribution. (Zeroed-out weights are hidden; only the survivors are plotted.)\n① dense — the familiar bell-shaped normal distribution. ② pruning (80%) — the small weights near 0 vanish wholesale. The middle is blown away and only the two tails remain. ③ retraining — the surviving weights re-fit to the data, becoming a clear bimodal distribution. ④ quantization (4-bit) — that continuous distribution folds into 16 discrete levels (this is what the K-Means in the [quantization article] does). Pruning empties out the middle of the horizontal axis (the small values), and quantization discretizes the remaining values into a few vertical lines. Since they touch different places, they stack without overlapping. As a result, the model size shrinks multiplicatively at each stage.\ndense 2,612 KB → pruning 604 KB (4.3×) → +quantization 147 KB (17.8×) → +Huffman 141 KB (18.6×) Meanwhile accuracy was actually held and improved, from 97.5% → 98.3% (thanks to retraining). So pruning alone gives 4.3×, adding quantization gives 17.8×, and Huffman on top gives 18.6×. The gains of each technique don\u0026rsquo;t add — they multiply. The original paper used this pipeline to shrink AlexNet by 35× and VGG-16 by 49× (with no accuracy loss). Data types → quantization → pruning: the three axes this series covered are ultimately orthogonal tools you can stack onto one model at the same time.\nRun it yourself Every plot above came from running the notebook below on Colab. Runtime → Run all is all it takes (for the 2:4 speed measurement, select an Ampere GPU such as an A100).\n📓 pruning.ipynb — OBD vs magnitude, retraining compression, 2:4 sparsity Repository: github.com/warpspaceinc/efficient-ml-practice Wrap-up What: random collapses quickly. Magnitude (|w|) is cheap and powerful, matching OBD\u0026rsquo;s second-order saliency — which is why it became the practical standard. How much: with repeated prune→retrain, even cutting 95% (20×) holds accuracy. How: unstructured only boosts the compression ratio. It takes 2:4 structured for the Ampere Sparse Tensor Core to actually accelerate — but only on large matrices. Together: pruning and quantization are orthogonal. Stack them the way Deep Compression does and the gains multiply (18.6× in our own measurement, with accuracy held). The insight Optimal Brain Damage planted in 1989 still holds. Only, for it to become a \u0026ldquo;real payoff,\u0026rdquo; the hardware 36 years later had to be ready to receive that sparsity. The history of pruning is, in a way, the history of algorithms and hardware waiting for each other.\nIn the next article we\u0026rsquo;ll cover how to set sparsity automatically, layer by layer (sensitivity analysis · AMC · NetAdapt).\nReferences Y. LeCun, J. S. Denker, S. A. Solla. Optimal Brain Damage. Advances in Neural Information Processing Systems (NeurIPS), 1989. S. Han, J. Pool, J. Tran, W. J. Dally. Learning both Weights and Connections for Efficient Neural Networks. NeurIPS, 2015. S. Han, H. Mao, W. J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. ICLR, 2016. A. Mishra et al. Accelerating Sparse Deep Neural Networks (2:4 Structured Sparsity). arXiv:2104.08378, 2021. S. Han et al. EIE: Efficient Inference Engine on Compressed Deep Neural Network. ISCA, 2016. M. Horowitz. Computing\u0026rsquo;s Energy Problem (and what we can do about it). ISSCC, 2014. Course reference: MIT 6.5940 TinyML and Efficient Deep Learning Computing (Song Han), Lec 3–4 Pruning \u0026amp; Sparsity. ","permalink":"http://blog.caveduck.io/posts/neural-net-pruning/","summary":"Starting from LeCun\u0026rsquo;s 1989 Optimal Brain Damage, this is the story of shrinking a model by cutting weights away (pruning). What to cut, how, and how much; the surprise of compression that survives even a 95% cut thanks to retraining; and the point that you only get a real payoff when the hardware (NVIDIA 2:4) backs you up — all with plots we measured ourselves on an MNIST MLP.","title":"Efficient ML #3 — Neural Net Pruning"},{"content":" This post is a follow-up to Data Types in the Deep Learning Era. If the previous post looked at how INT/FP data types interpret bits as numbers, this one actually takes an already-trained model and shrinks it with those low-bit types, then measures the result. All the code comes from the notebooks in github.com/warpspaceinc/efficient-ml-practice, and the plots below were produced by running that code as-is on Colab (T4).\nDo we really need all 32 bits? There was one thing we noticed at the end of the last post: a neural net\u0026rsquo;s weights are usually clustered in a bell shape around 0, with long tails. And indeed, if you train a small MNIST MLP (784→256→128→10) for just 2 epochs and plot the histogram of the first layer\u0026rsquo;s weights, that\u0026rsquo;s exactly what you get.\nIf the values are packed into such a narrow range, do we really need to store each one as a separate 32-bit float and multiply in 32 bits? Quantization answers \u0026ldquo;no.\u0026rdquo; In this post we run the two canonical approaches directly in code.\nK-Means quantization — group similar values under a shared representative to reduce storage (non-uniform). Linear quantization — map onto an integer grid so that both storage and compute become integer (uniform). Both apply to an already-trained model after the fact — this is PTQ (post-training quantization) — and here we only quantize the weights.\n1. K-Means Quantization — Sharing Representatives to Cut Storage The first idea comes from the 2016 Deep Compression paper1. The pitch is to lump similar weights together and cover them all with a single representative value. For example, 2.09, 1.92, 1.87 are all \u0026ldquo;roughly 2.0,\u0026rdquo; so we replace them with a single 2.00 and, at each position, just record an index for \u0026ldquo;which representative to use.\u0026rdquo;\nThe procedure has three steps.\nClustering — K-Means cluster all weights into $k = 2^N$ groups. Codebook — store each group\u0026rsquo;s centroid as its representative value. This list is the codebook. Indices — at each position, instead of a 32-bit float, store only an $N$-bit integer pointing to \u0026ldquo;which cluster it is.\u0026rdquo; In code we just use scikit-learn\u0026rsquo;s KMeans. To avoid local optima, we initialize with values that split the min–max range uniformly.\nfrom sklearn.cluster import KMeans import numpy as np def kmeans_quantize(w, n_bits, fit_sample=20000): \u0026#34;\u0026#34;\u0026#34;Quantize weights with K-Means. Returns the reconstruction, codebook, and indices.\u0026#34;\u0026#34;\u0026#34; shape = w.shape flat = w.reshape(-1, 1) k = 2 ** n_bits lo, hi = float(flat.min()), float(flat.max()) init = np.linspace(lo, hi, k).reshape(-1, 1) # linear initialization idx = np.random.choice(len(flat), min(fit_sample, len(flat)), replace=False) km = KMeans(n_clusters=k, init=init, n_init=1, max_iter=50).fit(flat[idx]) codebook = km.cluster_centers_.reshape(-1) # list of representatives indices = km.predict(flat) # each weight → cluster index recon = codebook[indices].reshape(shape) # reconstruction return recon, codebook, indices If you run it at 2 bits ($k=4$) and overlay the result on the original distribution, you can see the representatives (vertical lines) land densely around 0, where the values are packed. That\u0026rsquo;s what \u0026ldquo;non-uniform\u0026rdquo; means.\nHow much do we save? The compression ratio is bit-accounting. For $M$ parameters and $N$-bit quantization:\n$$\\text{original} = 32M \\text{ bit}, \\qquad \\text{compressed} = \\underbrace{N \\cdot M}_{\\text{indices}} + \\underbrace{32 \\cdot 2^N}_{\\text{codebook}} \\text{ bit}$$num = w.size; k = 2 ** N_BITS orig_bits = 32 * num comp_bits = N_BITS * num + 32 * k # indices + codebook print(f\u0026#34;compression: {orig_bits/comp_bits:.2f}x\u0026#34;) When $M \\gg 2^N$, the codebook term becomes negligible and the compression ratio converges to $32M/NM = \\mathbf{32/N}$×. That\u0026rsquo;s about 16× at 2 bits and 8× at 4 bits. (Measured compression ratio for the first layer fc1: 15.99×)\nAn important limitation: it only cuts storage This has to be stated clearly. What K-Means quantization reduces is only the size stored on disk/in memory. At inference time you still have to decode the indices through the codebook to reconstruct the original float weights before multiplying.\nStorage: integer indices (small) ✓ Compute: still floating-point arithmetic with the reconstructed 32-bit floats ✗ We saved memory bandwidth, but the multiplier (MAC) is still running floats. To actually speed up integer compute too, you need the next approach.\n2. Linear Quantization — Inference with Integer Arithmetic Only Linear quantization ties integers and reals together with a single-line affine mapping. It was proposed by Jacob et al. in 20182, and it\u0026rsquo;s exactly the method behind TensorFlow Lite\u0026rsquo;s INT8 quantization.\n$$r = S \\cdot (q - Z)$$ $r$ — the original real value $q$ — the quantized integer $Z$ — the zero point. The integer that maps exactly to real $0$. A device for representing common values like the 0 after ReLU, or padding, with no error. $S$ — the scale. How much one integer step is worth in real terms (floating-point). $S$ and $Z$ come from lining up the two ends of the real range $[r_{\\min}, r_{\\max}]$ with the two ends of the integer range $[q_{\\min}, q_{\\max}]$.\n$$S = \\frac{r_{\\max} - r_{\\min}}{q_{\\max} - q_{\\min}}, \\qquad Z = \\text{round}\\left(q_{\\min} - \\frac{r_{\\min}}{S}\\right)$$In code we only need to compute $S$ and $Z$; quantization (round·clamp) and reconstruction (dequant) are handled by PyTorch\u0026rsquo;s built-in fake_quantize_per_tensor_affine. Arbitrary bit widths are set via quant_min/quant_max.\nimport torch def linear_quantize(w, n_bits, symmetric=False): \u0026#34;\u0026#34;\u0026#34;Linear (affine) quantization via PyTorch\u0026#39;s built-in fake_quantize. Returns the reconstruction and S, Z.\u0026#34;\u0026#34;\u0026#34; t = torch.as_tensor(w, dtype=torch.float32) if symmetric: # Z=0 fixed (simpler compute) qmax = 2 ** (n_bits - 1) - 1; qmin = -qmax S = float(t.abs().max()) / qmax; Z = 0 else: # asymmetric: match the range exactly qmin = -2 ** (n_bits - 1); qmax = 2 ** (n_bits - 1) - 1 S = (float(t.max()) - float(t.min())) / (qmax - qmin) Z = int(round(qmin - float(t.min()) / S)) recon = torch.fake_quantize_per_tensor_affine(t, S, Z, qmin, qmax) return recon.numpy(), S, Z Run it at 2 bits and the representatives (vertical lines) land at uniform spacing $S$. This is exactly where it contrasts with K-Means — here the grid is uniform no matter where the data is packed.\nWhy does \u0026ldquo;integer arithmetic all the way\u0026rdquo; become possible? The real value of Linear quantization isn\u0026rsquo;t storage but compute. If you rewrite each value of the matrix product $Y = WX$ through the affine mapping:\n$$q_Y = \\frac{S_W S_X}{S_Y}\\big(q_W q_X - Z_W q_X - Z_X q_W + Z_W Z_X\\big) + Z_Y$$Everything inside the parentheses, including $q_W q_X$, is integer multiply-and-add, and terms independent of the input like $Z_W Z_X$ can be precomputed. The only real quantity is the leading scale ratio $\\frac{S_W S_X}{S_Y}$, and empirically this value is always in $(0,1)$, so it\u0026rsquo;s handled as a fixed-point multiply + bit shift of the form $2^{-n} M_0$. In the end no floating-point unit is needed at all — the \u0026ldquo;integer arithmetic is tens of times cheaper than float\u0026rdquo; from the energy table in Part 1 becomes reality right here.\nSince weights are usually symmetric about 0, in practice we often use symmetric quantization with $Z_W = 0$. Then all the $Z_W$-related terms vanish outright and the equation gets even cleaner.\n3. So, How Much Do We Lose? — Measure It Directly Words alone don\u0026rsquo;t give a feel for it, so we compared the three methods (Linear asymmetric / Linear symmetric / K-Means) across bit widths using MNIST test accuracy. We quantized the weights of all three Linear layers of the model (replacing them with the reconstructions) and then measured accuracy.\ndef eval_linear(model, n_bits, symmetric): m = copy.deepcopy(model) with torch.no_grad(): for layer in [m.fc1, m.fc2, m.fc3]: wl = layer.weight.detach().cpu().numpy() recon, *_ = linear_quantize(wl, n_bits, symmetric=symmetric) layer.weight.copy_(torch.tensor(recon, dtype=torch.float32, device=device)) return accuracy(m) bits_list = [2, 3, 4, 8] acc_asym = [eval_linear(model, b, False) for b in bits_list] acc_sym = [eval_linear(model, b, True) for b in bits_list] # K-Means compared the same way via eval_kmeans(model, b) bits Linear (asym) Linear (sym) K-Means float32 8 97.2% 97.2% 97.2% 97.2% 4 97.1% 97.0% 97.1% 97.2% 3 96.7% 94.4% 96.8% 97.2% 2 58.1% 18.8% 93.9% 97.2% The big picture to take away is this.\nAt 8 bits it\u0026rsquo;s essentially free. All three methods hold accuracy so well you can\u0026rsquo;t tell them apart from float32. The lower the bit width, the more K-Means wins. Because it places representatives to match the data distribution (non-uniform), at the same 2 bits its error is smaller than Linear\u0026rsquo;s uniform grid. This is even clearer in the MSE curve below. But accuracy isn\u0026rsquo;t everything. Even if K-Means has smaller error at 2 bits, its inference compute is still float. Linear accepts a bit more error in exchange for making the compute itself integer, which is what actually speeds things up on real hardware. The two methods aren\u0026rsquo;t competitors — they\u0026rsquo;re tools for different purposes.\nMethod Storage Compute Grid Original FP weights FP arithmetic — K-Means quantization integer indices + FP codebook FP arithmetic non-uniform Linear quantization integer weights integer arithmetic uniform Try It Yourself All the plots and tables above came from running the two notebooks below on Colab. Runtime → Run all is all it takes, and you can apply it to your own model just by uploading its .safetensors and changing the path.\n📓 linear-quantization.ipynb — affine mapping, symmetric/asymmetric 📓 kmeans-quantization.ipynb — codebook/indices, compression ratio Wrap-up Quantization = reducing the number of representable values. Since weights are packed around 0, using all 32 bits is wasteful. K-Means (non-uniform) places representatives to match the distribution, cutting storage but keeping compute in float. Its error is small at low bit widths. Linear (uniform) turns both storage and compute into integers via the affine mapping $r=S(q-Z)$. It\u0026rsquo;s the foundation of hardware acceleration. 8 bits is nearly free; push all the way to 2 bits and the choice of method decides your accuracy. In the next experiment note, we\u0026rsquo;ll take on activation quantization and QAT (quantization-aware training), which we couldn\u0026rsquo;t cover here, and build the integer inference pipeline all the way to the end.\nHan, Mao, Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. ICLR 2016.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nJacob et al. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR 2018.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"http://blog.caveduck.io/posts/neural-net-quantization/","summary":"We take the trained 32-bit weights of a neural net and shrink them down to 2–8 bits using two approaches: K-Means (non-uniform) and Linear (uniform, integer arithmetic). From the affine mapping r=S(q−Z) all the way to the compression-ratio vs. accuracy trade-off, with plots measured by actually running the code on an MNIST MLP.","title":"Efficient ML #2 — Neural Net Quantization"},{"content":" Warpspace runs a series of experiments to make models smaller so we can run Caveduck.io on-device and at low cost. This is the first post (#1, data types) of that Efficient ML series. The follow-ups - #2 Quantization and #3 Pruning - take the data types covered here, shrink an actual model, and measure the results firsthand.\nWhy did we end up with so many data types? Just a few years ago, float32 alone was enough for all the numbers in deep learning. But look at papers or model cards these days and you get a flood of data types: FP16, BF16, FP8 (E4M3/E5M2), INT8, INT4, FP4, NF4… From the names alone it\u0026rsquo;s hard to tell what\u0026rsquo;s what, or why there are so many.\nThe reason is simple: low-bit arithmetic is just cheap. On a 45nm process, the energy cost per operation looks roughly like this.\nOperation Energy (pJ) 8-bit int ADD 0.03 32-bit int ADD 0.1 32-bit float ADD 0.9 8-bit int MULT 0.2 32-bit float MULT 3.7 An 8-bit integer multiply is about 18x cheaper than a 32-bit float multiply, and addition is 30x cheaper. On top of that, the cost of reading a single value from memory is hundreds of times more expensive than one multiply or add. So for the same model, the fewer bits you pack a number into, the more power, memory, and bandwidth you save. All those data types above are ultimately different answers to the question: \u0026ldquo;how many bits, and where do we spend them?\u0026rdquo;\nThe catch is that fewer bits means fewer distinct numbers you can represent, and you lose that much precision or range. So each format\u0026rsquo;s character comes down to how it splits its bits among sign, exponent, and fraction. This post takes apart those allocation rules — how a bit pattern actually gets interpreted as a number — one at a time.\nFor each data type, I\u0026rsquo;ve included a widget where you can click the bits directly. Press a 0 or 1 to flip it and the formula and result update in real time. Rather than reading an explanation with your eyes, actually poking at the bits will give you a much faster feel for it.\n1. Integers Unsigned Integer The simplest one. Each of the $n$ bits carries a place value of $2^k$, and you sum the place values of every bit that\u0026rsquo;s on ($=1$).\n$$\\text{value} = \\sum_{i=0}^{n-1} b_i \\cdot 2^i$$The representable range is $[0,\\ 2^n - 1]$. For 8 bits, that\u0026rsquo;s 0 to 255.\nTry pressing the bits in the widget below. The default 00110001 is $2^5 + 2^4 + 2^0 = 49$.\nSigned Integer To represent negative numbers, you need a sign. There are two approaches.\nSign-Magnitude — use the leading bit as the sign ($0$=positive, $1$=negative) and the rest to represent the magnitude. Intuitive, but it has a fatal flaw: 00000000 and 10000000 are both zero (+0 and −0). Two zeros make the hardware more of a hassle. The range is $[-2^{n-1}+1,\\ 2^{n-1}-1]$.\nTwo\u0026rsquo;s Complement — what modern computers actually use. The idea is simple: make the leading bit\u0026rsquo;s place value negative. That is, the MSB\u0026rsquo;s weight is $-2^{n-1}$ rather than $+2^{n-1}$.\n$$\\text{value} = -b_{n-1}\\cdot 2^{n-1} + \\sum_{i=0}^{n-2} b_i \\cdot 2^i$$This way there\u0026rsquo;s only one zero, 00000000, and the range $[-2^{n-1},\\ 2^{n-1}-1]$ reaches one step further on the negative side. Below, 11001111 is $-2^7 + 2^6 + 2^3 + 2^2 + 2^1 + 2^0 = -49$. Try turning off the MSB and see how it changes.\n2. Fixed-Point Integers alone can\u0026rsquo;t hold fractional values. The easiest extension is to fix the position of the decimal point. You just read the bit string as a two\u0026rsquo;s complement integer, then scale it by a fixed $2^{-f}$.\n$$\\text{value} = (\\text{value read as integer}) \\times 2^{-f}$$Below, 8 bits are split into a 4-bit integer part and a 4-bit fractional part ($f=4$). Each bit\u0026rsquo;s weight runs $2^3, 2^2, \\dots, 2^0, 2^{-1}, \\dots, 2^{-4}$. Read as an integer, 00110001 is 49, and $49 \\times 2^{-4} = 3.0625$.\nFixed-point is simple but its limits are clear. Because the decimal point is fixed, the range of magnitudes it can represent is narrow. It struggles to handle very large and very small numbers at the same time. Solving that problem is the job of our next protagonist: floating-point.\n3. Floating-Point — IEEE 754 Floating-point is exactly what the name says: the decimal point floats. Think of it as the binary version of scientific notation ($1.5 \\times 10^3$). The bits are split into three parts.\nSign — 1 bit Exponent — sets the scale of the magnitude → determines the range Fraction — sets the fine detail → determines the precision The formula for interpreting a normal number is this.\n$$\\text{value} = (-1)^{\\text{sign}} \\times (1 + \\text{Fraction}) \\times 2^{\\text{Exponent} - \\text{bias}}$$Here bias is an offset that lets the exponent represent negative values too, equal to $2^{e-1}-1$ ($e$ = number of exponent bits). And note the $(1 + \\cdots)$ in front of the fraction: a normal number always has an implicit 1 at the front (this is called the \u0026ldquo;implicit leading 1\u0026rdquo;).\nHold on — a note on terminology: Fraction / Mantissa / Significand\nThe terminology here gets mixed up a bit across the literature, so it\u0026rsquo;s easy to confuse. The proper name for the whole $1.\\text{Fraction}$ in the formula above — the significant-digits part — is the Significand.\nFraction — refers only to the fractional part actually stored in the bits. The bits shaded yellow in the widget are exactly this. It\u0026rsquo;s the sub-decimal value $0.\\text{b}_1\\text{b}_2\\cdots$. Significand — the whole $1.\\text{Fraction}$ including the implicit 1. This is the significant value actually multiplied in. Mantissa — historically the term for the fractional part of a logarithm table, but in floating-point it\u0026rsquo;s usually used loosely to mean the same thing as Fraction. The IEEE 754 standard itself uses \u0026ldquo;Significand\u0026rdquo; as the official term and discourages \u0026ldquo;mantissa,\u0026rdquo; but in the field \u0026ldquo;mantissa\u0026rdquo; is still common. To summarize: Significand = 1 + Fraction, and when people casually say \u0026ldquo;mantissa\u0026rdquo; they usually mean the stored Fraction. In this post we\u0026rsquo;ll call the field stored in the bits the Fraction.\nBelow is 32-bit single precision (FP32). Sign 1 + exponent 8 + fraction 23 = 32 bits, with bias 127. The default represents $0.265625 = (1 + 0.0625) \\times 2^{125-127}$. Try pressing the exponent bits one at a time and watch the value jump by a factor of 2 each time.\nSpecial values: 0, infinity, NaN, and subnormals There\u0026rsquo;s one odd thing about the formula. Because of $(1 + \\text{Fraction})$, a normal number can\u0026rsquo;t represent 0. So IEEE 754 uses the exponent field as a special signal.\nExponent Fraction = 0 Fraction ≠ 0 Interpretation 00…0 (=0) $\\pm 0$ subnormal $(-1)^s \\times \\text{Fraction} \\times 2^{1-\\text{bias}}$ 00…1 ~ 11…0 normal normal $(-1)^s \\times (1+\\text{Fraction}) \\times 2^{\\text{Exp}-\\text{bias}}$ 11…1 (=max) $\\pm\\infty$ NaN — The key is when the exponent is all zeros. In that case you drop the implicit 1 ($1+\\text{Fraction} \\to \\text{Fraction}$) and fix the exponent at $2^{1-\\text{bias}}$. What this produces are subnormal (denormal) numbers, which densely fill in the very small values near 0. Conversely, when the exponent is all ones, you get infinity (fraction 0) and NaN (fraction ≠ 0).\nTry building these yourself in the FP32 widget above.\nSet the exponent to all ones (0 11111111 0…0) → +∞ From there, turn on any fraction bit → NaN All zeros → 0 Leave the exponent at 0 and turn on a few fraction bits → a tiny subnormal value The wider the exponent, the wider the range; the wider the fraction, the higher the precision. Exponent → Range, Fraction → Precision. This one line is the core trade-off behind the design of every low-precision format that follows.\n4. Half the size: FP16 and BF16 FP32 is accurate, but it eats a full 32 bits. Deep learning often doesn\u0026rsquo;t need that much precision, so we use 16-bit formats. Same 16 bits, but the split comes down to where you allocate them.\nFP16 (IEEE 754 Half Precision) Exponent 5 + fraction 10. Bias 15. An allocation that invests more in precision (the fraction). Below is $1\\,10001\\,1100000000$, i.e. $-(1+0.75)\\times 2^{17-15} = -7.0$.\nBF16 (Google Brain Float) Exponent 8 + fraction 7. Bias 127. It has the same total bit count as FP16, but keeps the same 8-bit exponent as FP32. That means its range is identical to FP32, at the cost of precision. It\u0026rsquo;s popular because it doesn\u0026rsquo;t have to worry about overflow in situations where the scale of values swings wildly, like gradients during training.\nBelow is $2.5 = (1 + 0.25)\\times 2^{1}$ represented in BF16 ($0\\,10000000\\,0100000$).\nGet a feel for the difference in exponent bit count across the two widgets. BF16 has 8 exponent cells, so it reaches very large/small numbers, but with only 7 fraction cells its values are sparse. FP16 is the opposite.\n5. Going lower: FP8 and FP4 FP8 (E4M3 / E5M2) 8-bit floating-point is supported by the latest hardware (e.g. Nvidia Hopper/Blackwell). Two allocations are used almost like standards.\nE4M3 — exponent 4 + fraction 3. Precision-first. Mainly used for weights and activations in the forward pass. It has no INF and uses only S.1111.111 as NaN. The largest representable normal value is $448$. E5M2 — exponent 5 + fraction 2. Range-first. Used for large-scale values like gradients in the backward pass. It has INF and NaN like IEEE. Starting with E4M3. Bias 7. The default 0 0111 000 is $(1+0)\\times 2^{7-7} = 1.0$.\nE5M2. Bias 15. With 5 exponent cells it holds a much wider range, but with only 2 fraction cells the spacing between values is coarse.\nINT4 and FP4 The extreme end. With 4 bits, you can represent only 16 values. How those 16 are laid out differs from format to format.\nINT4 — two\u0026rsquo;s complement integer. Placed at uniform spacing from $-8$ to $7$.\nFP4 distributes its values differently depending on the exponent/fraction split. The more exponent bits, the more the values pack in near 0 and spread out toward the edges, i.e. non-uniformly.\nE1M2 — exponent 1 + fraction 2. Closest to an integer (bias 0). 0111 = $(1+0.75)\\times 2^{1-0} = 3.5$. E2M1 — exponent 2 + fraction 1 (bias 1). 0111 = $(1+0.5)\\times 2^{3-1} = 6$. E3M0 — exponent 3 + fraction 0 (bias 3). With no fraction, it effectively represents only powers of 2. 0111 = $(1+0)\\times 2^{7-3} = 16$. Press the random button and watch the values spread out exponentially, like $\\dots, 4, 8, 16$. Plotting the \u0026ldquo;16 representable values\u0026rdquo; of these four formats on a single number line makes the differences in character jump right out. INT4 is evenly spaced; FP4 packs values near 0 and spreads them at the edges the more exponent bits you add.\nHere\u0026rsquo;s one interesting observation. The weight distribution of a neural network is usually clustered near 0 with long tails. If so, then rather than the INT approach of laying out values uniformly, an FP-style format that densely fills the region near 0 can fit the actual values better with the same 4 bits. Which format suits which distribution — that\u0026rsquo;s why choosing a data type leads directly to accuracy. (We measure this observation on a real model in the quantization part.)\nWrapping up That\u0026rsquo;s the story of what the data types you run into in deep learning actually mean. To summarize:\nInteger / fixed-point — values at uniform spacing. Simple, but narrow range. Floating-point — split its bits between the exponent for range and the fraction for precision. Dense near 0, sparse at the edges. Reducing bits = reducing the number of representable values. From FP32\u0026rsquo;s roughly 4.3 billion down to FP4\u0026rsquo;s 16. Now when you see names like FP16, BF16, FP8 E4M3, or INT4, you\u0026rsquo;ll be able to picture the bit layout in your head. In deep learning, \u0026ldquo;which data type should I use?\u0026rdquo; is no longer a trivial implementation detail but a design choice that decides how small and fast — yet how accurate — you can run a model.\nIn the next post, we take these data types as our weapons and actually shrink an already-trained 32-bit model down to 2–8 bits — running the code ourselves to measure how much accuracy is preserved and how much the size shrinks.\n👉 Neural Net Quantization: Shrinking a 32-bit model down to 2 bits\n","permalink":"http://blog.caveduck.io/posts/deep-learning-data-types/","summary":"INT8, FP16, BF16, FP8 (E4M3/E5M2), FP4 — what do all these data types flooding deep learning model cards actually mean, and how do bits get interpreted as numbers? We take them apart one by one, with a widget where clicking bits updates the formula and value in real time.","title":"Efficient ML #1 — Deep Learning Data Types"},{"content":"Hello World! Welcome to the official Warpspace Blog! We\u0026rsquo;re excited to launch this space where we\u0026rsquo;ll share our journey, insights, and experiences in building Caveduck.io - your AI companion service.\nWhat to Expect On this blog, you\u0026rsquo;ll find:\nTechnical Deep Dives: Behind-the-scenes looks at the technology powering Caveduck Product Updates: New features, improvements, and roadmap insights AI \u0026amp; Technology Trends: Our perspectives on the evolving AI landscape Team Stories: Meet the people building the future of AI companions Stay Connected Follow us on social media and subscribe to our RSS feed to stay updated with our latest posts.\nWe\u0026rsquo;re thrilled to have you here. Let\u0026rsquo;s build the future together!\n- The Warpspace Team\n","permalink":"http://blog.caveduck.io/posts/welcome/","summary":"Introducing the official Warpspace Blog - where we share our journey building Caveduck.io","title":"Welcome to Warpspace Blog"},{"content":"About Warpspace Inc. Warpspace Inc. is the company behind Caveduck.io, an innovative AI companion service designed to provide meaningful and engaging interactions.\nOur Mission We believe in creating AI companions that are helpful, safe, and genuinely enjoyable to interact with. Our goal is to push the boundaries of what\u0026rsquo;s possible in AI-human interaction while maintaining the highest standards of quality and user experience.\nCaveduck.io Caveduck.io is our flagship product - an AI companion service that offers personalized, engaging conversations. Whether you\u0026rsquo;re looking for a creative collaborator, a learning partner, or simply someone to chat with, Caveduck is here for you.\nContact Email: contact@caveduck.io Website: caveduck.io Company: warpspace.caveduck.io GitHub: github.com/warpspaceinc ","permalink":"http://blog.caveduck.io/page/about/","summary":"About Warpspace Inc. and Caveduck.io","title":"About"}]