That Free Korean NPU the Government Hands Out — Can You Actually Use It? Notes From Porting an Unsupported Model

Background

We ran an experiment to move a persona-deviation detector — a binary classifier that catches when an AI character breaks persona and answers like a generic assistant or refuses a request — that we had been serving on GPU in production onto a Korean NPU. Bottom line up front: after picking apart the causes one by one in the face of the “officially unsupported” wall, we got the port working. On-device inference clocked in at 9.7ms (fixed 512 tokens, near-zero jitter) — latency and stability that hold up well against our existing cloud-GPU serving. This post is an in-order record of what we tried, where we got stuck, and how we solved it.

First, some context on how we got access to a Korean NPU. Korea supplies domestic AI chips through multiple government-led channels. This isn’t marketing copy — there really are several avenues through which a startup or researcher can obtain Rebellions’ ATOM family, either free or as a pilot.

• K-Cloud Project (MSIT · NIPA, 3 years from 2023) — the “AI Semiconductor Farm construction & demonstration” program. Three CSPs (Naver Cloud, KT Cloud, NHN Cloud) and three AI-chip makers (Rebellions, Sapeon, FuriosaAI) form a consortium to build and validate domestic-NPU-based high-performance cloud. (ZDNet, NIPA notice)
• High-Performance Computing Support Program (MSIT · NIPA, ~16.7B KRW in 2026) — provides domestic NPU servers free of charge to selected demand companies. Several supply companies were picked (KT Cloud, Gabia, etc.); Gabia, for instance, provides Rebellions ATOM-Max at no cost. (Etoday, VentureSquare)

So through combinations of CSP (KT/NHN/Naver/Gabia…) × program (farm pilot / free provisioning), ATOM, ATOM+, and ATOM-Max are being distributed through a variety of windows. We too obtained 4× ATOM+ cards through one of these paths, in the form of KT Cloud’s AI Nexus (Backend.AI-based under the hood).

We expected three gains from this port: ① cost savings on always-on GPU serving (government-provided hardware), ② lower, more stable inference latency, and ③ a vendor option beyond NVIDIA GPUs.

The problem was what came next.

The stack at a glance — from software to hardware

Before diving in, let’s lay out the layers this work sits on. Turning a single line of text into a verdict passes through several layers running from software (top) to hardware (bottom). (Definitions of each term are collected in the Glossary at the bottom.)

The crux is the compiler layer. On a GPU this stack is mostly transparent and you can run the model as-is, but on an NPU the compiler becomes the gatekeeper. Most of this post is ultimately the story of getting through this one layer.

A quick aside — what does “compile” mean here?

To keep going with the NPU story, we need to pin down what “compiling a model” means. In short, it’s translating a PyTorch model ahead of time into a form a specific accelerator can execute directly. The GPU default, eager execution (invoking a kernel on the fly for each line of Python), is flexible but has spiky latency; compilation freezes the entire graph into a single deterministic binary, yielding low latency and low jitter. On an NPU this compilation is effectively mandatory, and everything later — having to disable reference_compile or dynamic-shape branches — all stems from this “freeze the graph ahead of time” constraint.

There’s a separate post that walks through the concept itself — eager vs. compile, graph capture, op fusion, kernel lowering (OpenAI Triton, etc.), and static shapes — from the ground up with figures and an interactive demo → Neural Net Compilation, Made Simple. Here we only touch on as much as our case requires.

Recognizing the problem — an NPU must be compiled ahead of time

A GPU can run a PyTorch model as-is, but an NPU is different. You have to compile ahead of time for “this model, this input size” and convert it into an NPU-specific binary. Handling that conversion is Rebellions’ compiler rebel-compiler, with the integration layer optimum-rbln on top, bridging standard model definitions to it.

But our backbone was mmBERT, i.e. the ModernBERT architecture. As it turned out, it was missing from the supported list.

• The encoders officially supported by optimum-rbln are only bert / distilbert / roberta / xlm-roberta. ModernBERT is not supported.
• No ModernBERT variant in the Rebellions Model Zoo either.

The usual move at this point is to switch to a supported model like XLM-R. But our model’s accuracy was already validated on mmBERT, and swapping the backbone carries a heavy re-training and re-validation cost. So we decided to implement support directly.

Hypothesis — “is support really impossible?”

Look into what the “unsupported” label actually amounts to, and only two kinds of things really block you: (1) there’s an op the compiler doesn’t understand, or (2) the integration layer lacks a wrapper for that model.

We analyzed the elements considered to be ModernBERT’s novel additions one by one.

Every one of them was an element already present in a supported decoder model (Gemma2/3, etc.). In other words, the compiler’s op compatibility is effectively proven, and the real difference is only “encoder vs. decoder.” And the encoder base class (RBLNModelForMaskedLM) already existed. In the end it was problem (2): only the wrapper was missing.

One thing to watch was the attention implementation. ModernBERT defaults to Flash Attention 2 (HuggingFace’s flash_attention_2 backend), which is an NVIDIA-CUDA-only kernel that rebel-compiler can’t compile for the NPU. Fortunately, forcing PyTorch’s standard abstraction API SDPA (scaled_dot_product_attention) lets the compiler recognize it and translate it into its own NPU kernel. It amounts to swapping only the backend, with no accuracy loss.

Visually, in the computation graph of the mmBERT-base (ModernBERT architecture) we were porting, only two spots were blocking. The rest of the ops (RoPE, GeGLU, etc.) are already supported by the compiler (green).

flowchart TB
    IDS["input_ids · [1,512]"] --> EMB["Token Embedding + LayerNorm"]
    MASK["attention_mask · [1,512]"] -.-> ATTN
    EMB --> LN1
    subgraph LYR["Encoder layer × 22 (mmBERT-base = ModernBERT)"]
      direction TB
      LN1["LayerNorm (pre-norm)"] --> ROPE["RoPE (global θ / local θ)"]
      ROPE --> ATTN["Attention<br/>local (sliding window) · global (every 3rd layer)"]
      ATTN --> RES1["＋ residual"]
      RES1 --> LN2["LayerNorm"] --> MLP["GeGLU MLP"] --> RES2["＋ residual"]
    end
    RES2 --> FLN["final LayerNorm"] --> POOL["mean pooling"] --> HEAD["linear head → logits(2)"]
    ATTN -.- NOTE1["⚠ Problem ① Flash Attention 2 (CUDA-only)<br/>→ replace with SDPA · disable sliding_window_mask·unpadding (dynamic shape)"]
    LYR -.- NOTE2["⚠ Problem ② reference_compile=False<br/>(remove torch.compile inside forward)"]
    class ATTN problem
    class ROPE,MLP okc
    class NOTE1,NOTE2 note
    classDef problem fill:#7f1d1d,stroke:#f87171,stroke-width:3px,color:#fff
    classDef okc fill:#14532d,stroke:#4ade80,color:#fff
    classDef note fill:#3a2f0b,stroke:#e0a020,color:#fbe8a6

In other words, it wasn’t the whole architecture that was the problem — only two spots collided with NPU compilation: the attention kernel (FA2) and the torch.compile inside forward (reference_compile). Stripping these two out in the adapter was the next step.

Writing the adapter — three things to control

We wrote the ModernBERT adapter by referencing the pattern of the sibling class RBLNRobertaForMaskedLM. There are three points to control.

1. Disable incompatible options at the config level — attn_implementation="sdpa", reference_compile=False (ModernBERT calls torch.compile inside forward, which breaks trace/export).
2. Block incompatible kwargs at the wrapper level — explicitly disable unpadding/sequence-packing args (indices, cu_seqlens, sliding_window_mask, etc.) to remove dynamic-shape branches.
3. Simplify the input signature — just input_ids + attention_mask.

Up to here you can verify without an NPU. Extracting the graph on a MacBook with torch.jit.trace and torch.export, the error vs. eager across two different inputs was max|diff| = 0.0 — a numerically identical graph. Once you’ve confirmed this much, you’re confident the port is feasible. All that’s left is real-hardware validation.

On-hardware compile — the first try failed

We attached to an ATOM+ session (the container ships rebel-compiler and optimum-rbln pre-installed, so no separate install was needed) and ran the compile. The first attempt failed with this error:

ValueError: Configuration for RBLNModernBertForMaskedLMConfig not found.

Tracing the cause: the optimum-rbln we had referenced was the latest main, but what was actually installed in the container was the 0.10.2 release, and that version resolved config classes only via a name-based registry (a top-level attr on optimum.rbln, or the internal CONFIG_MAPPING). Our custom adapter’s config wasn’t registered there, so it failed as “not found.” A trap created by SDK version drift.

The fix was one line — pin the config class directly on the model class so it skips the registry lookup:

class RBLNModernBertForMaskedLM(RBLNModelForMaskedLM):
    _rbln_config_class = RBLNModernBertForMaskedLMConfig  # bypass the registry

Rerun, and this time it passed:

[rebel-compiler] Target NPU: RBLN-CA22
[rebel-compiler] Tensor parallel size: 1
Computation graph generation   ████████████ 100%
Computation graph optimization ████████████ 100%
saved -> compiled_model.rbln

The physical device confirmed via rbln-smi was RBLN-CA22 (ATOM+), 15.7GiB. The moment an “unsupported” model compiled on a Korean NPU. The compile itself finished in a dozen-odd seconds.

Applying it for real — porting the whole classifier

So far we had only compiled the backbone (MaskedLM). What we actually serve is a persona-deviation classifier with mean pooling + a linear head layered on top of it. Since that’s a custom nn.Module rather than a standard HuggingFace architecture, it was cleaner to use the compiler’s low-level API directly instead of optimum-rbln’s high-level path.

flowchart TD
    T["Raw text"] --> TOK["Tokenizer<br/>input_ids · attention_mask · [1, 512]"]
    TOK --> BB
    subgraph G["Compile unit — rebel.compile_from_torch ⇒ compiled_model.rbln (runs on ATOM+ NPU)"]
        direction TB
        BB["mmBERT backbone · SDPA"] --> MP["mean pooling"]
        MP --> LH["linear head"]
    end
    LH --> LOG["logits (2)"]
    LOG --> P["softmax → deviation probability"]

The tokenizer runs on CPU, and below it the backbone + pooling + head are bundled into a single compile unit that runs on the NPU. A single call to rebel.compile_from_torch(model, input_info=[("input_ids",[1,512],"int64"), ("attention_mask",[1,512],"int64")]) compiled these three into a single graph.

compiled_model.rbln is a serialized binary, so you can’t just open it up, but if you unfold the graph the compiler handles internally, it looks roughly like this (conceptual representation).

# compiled_model.rbln — internal graph (conceptual representation)
target: RBLN-CA22 (ATOM+)   tensor_parallel: 1   dtype: fp16
in  %input_ids      : i64[1, 512]
in  %attention_mask : i64[1, 512]

  %emb  = rbln.embedding       %input_ids                    -> f16[1,512,768]
  %h0   = rbln.layernorm       %emb
  # × N encoder layers (attention → mlp):
  %a    = rbln.sdpa_attention  %h{i}, mask=%attention_mask   @kernel=attn_fused_ca22
  %m    = rbln.geglu_mlp       %a                            @kernel=geglu_fused_ca22
  ...
  %enc  = rbln.layernorm       %hN                           -> f16[1,512,768]
  %pool = rbln.mean_pool       %enc, mask=%attention_mask    -> f16[1,768]
  %log  = rbln.linear          %pool, W=const[768,2]         -> f16[1,2]
out %log

The compile steps described earlier show up plainly here — the ops were lowered into ATOM+-specific kernels of the form @kernel=..._ca22, and every tensor shape is statically fixed to [1,512,...]. And the most important check — do the CPU and NPU predictions agree?

The numerical error at the logits level was at most ~4e-2 (a characteristic of NPU FP16 arithmetic), but the probabilities and decisions matched to four decimal places. The predictions reproduced exactly.

Benchmark

Same model, batch=1, fixed 512 tokens.

As impressive as the numbers was the stability of the distribution. p50 9.66ms / p99 9.73ms / max 10.6ms — essentially no jitter. Because it runs a fixed compiled graph deterministically, the latency doesn’t spike and stays flat. It was comfortably competitive even against the cloud-GPU serving we had been using. Power draw is also around 85W per card.

Did the expectations hold up?

To sum up, of the three, low latency and stability were confirmed beyond expectations (9.67ms, near-zero jitter). The vendor option opened up too, demonstrating that “even an unsupported model can be ported.” The serving configuration for cost savings is currently in preparation.

Lessons

• “Unsupported” usually means “no wrapper.” If the model is built from a combination of ops the compiler understands, you can port it with a thin adapter.
• The power of SDPA standardization. Drop down to PyTorch’s standard abstraction API instead of a vendor-locked kernel (FA2), and the backend (GPU or NPU) translates it into its own kernel. This is the crux of portability.
• Verify as much as possible locally, then go to hardware. Confirming graph equivalence (zero error) via trace/export locally drastically shrinks the surface you have to debug on real hardware.
• Watch out for SDK version drift. The internal API of the docs/latest-main and the actually-deployed release can differ. This time, a single line about the config-registry approach blocked the first compile.

There are probably plenty of teams leaving a Korean NPU in a “received but can’t use it” state. It wasn’t without trial and error, but at least for encoder-family models it can definitely be ported. Now that the government has laid out the hardware through so many channels, it’s worth a serious attempt.

Glossary

• NPU (Neural Processing Unit) — a chip specialized for neural-network computation. Unlike a general-purpose GPU, you fix the model to run and the input size ahead of time, compile it, and run it as a dedicated binary.
• Encoder model / ModernBERT / mmBERT — a neural net that converts text into a vector representation (understanding-centric, not generation). ModernBERT is the latest BERT-family architecture, and mmBERT is its multilingual variant.
• Compile (in the NPU context) — the process of converting a PyTorch model into an NPU-specific execution binary (.rbln) fitted to a specific input shape. Contrasts with a GPU’s immediate execution.
• rebel-compiler / optimum-rbln / rebel.Runtime — respectively, Rebellions’ compiler, the integration layer that connects a HuggingFace model to the compiler, and the runtime that executes the compiled binary on the NPU.
• SDPA vs. Flash Attention 2 — both are attention implementations. The FA2 here means HuggingFace’s flash_attention_2 backend (the NVIDIA-CUDA-only flash-attn kernels), which rebel-compiler can’t compile. SDPA (scaled_dot_product_attention) is a PyTorch standard abstraction API that each backend can translate into its own kernel, so it’s highly portable. (Note: RBLN also has its own flash_attn option for decoder KV-cache attention, but that’s different from HF’s FA2 backend and doesn’t apply to encoders.)
• RoPE / GLU·GeGLU — respectively, an embedding technique that injects positional information via rotation, and an MLP structure with gating applied. Both are widely used in modern transformers and are already supported by the Rebellions compiler.
• MaskedLM — the base form of an encoder (backbone) trained to mask part of a sentence and restore it. You put a pooling/classification head on top to make it a classifier.
• mean pooling — a method that averages per-token vectors to summarize a sentence into a single vector.
• ATOM+ / RBLN-CA22 — Rebellions’ NPU card (ATOM+) and its internal target identifier (RBLN-CA22).

References

• K-Cloud Project / AI Semiconductor Farm construction & demonstration — ZDNet, NIPA integrated notice
• 2026 High-Performance Computing Support Program (free domestic-NPU provisioning) — Etoday, VentureSquare
• Rebellions optimum-rbln (the integration layer the adapter is based on) — GitHub
• Rebellions Model Zoo — GitHub
• Rebellions RBLN SDK docs — docs.rbln.ai