How Is LFM3's Tri-Hybrid Architecture Built and Trained?

How is LFM3's tri-hybrid architecture built and trained?

LFM3-1.7B-Base is a 1.73B-parameter, 24-layer text model that mixes three layer types: 13 gated short convolutions, 5 sliding-window attention layers and 6 global attention layers. Liquid AI pre-trained it on 22T tokens in stages. A main pre-training run covers web, synthetic, code, math and science data. Then 3.5T tokens of curated mid-training data, heavy on agentic traces and tool calls, stretch the context window to 256K. Post-training turns the base checkpoint into LFM3-1.7B.

What are the three layer types in the tri-hybrid design?

The model card lists the stack as 13 gated short convolution layers, 5 sliding-window attention layers and 6 global attention layers. That is 24 layers total.

Each type does a different job. This is general background, not something the card spells out layer by layer:

• Gated short convolutions mix information across a small local neighborhood of tokens. Their cost per token stays flat as the sequence grows, and they don't need a key-value cache the way attention does.
• Sliding-window attention lets each token attend to a bounded window of recent tokens. This gives finer local reasoning than a convolution while still capping memory.
• Global attention lets any token attend to the whole context. It is the most expensive layer type, so the design uses it in only a quarter of the stack.

The model card does not give the window size or the order in which the layers are interleaved, so treat those as unknowns until you read the code or the release blog post. The code ships with the checkpoint and loads with trust_remote_code=True.

The practical logic is straightforward. If only 6 of 24 layers keep a full-context cache, long sequences cost much less memory than in an all-attention model of similar depth. That matters for a model aimed at phones and laptops.

What are the core specs?

At 4-bit precision, Liquid AI says the model takes about 1 GB of memory, so it fits on any phone or laptop.

How was LFM3-1.7B-Base pre-trained?

Pre-training runs in two broad stages: main pre-training and mid-training. The training recipe diagram on the model card breaks the 22T tokens into smaller steps.

Those figures add up to roughly the 22T total. The card states that the last 3.5T tokens are the curated mid-training portion, which matches the 3T plus 500B plus 50B at the end of the diagram.

Compared with LFM2, Liquid AI says the main pre-training mix adds more scientific papers, repository-level code, math and a wider range of knowledge sources. Repository-level code matters for long-context work, because the model sees how files in a project relate rather than isolated snippets.

Why train context length in stages?

Attention cost grows quickly with sequence length, so training on very long sequences from the start is expensive. The recipe instead spends most tokens at short context (4K for the code, math and science stage) and moves up in steps: 32K during mid-training, then 128K and 256K. The last two steps use far fewer tokens, 500B and 50B.

This staging pattern is common across long-context models. Most of the learning happens cheaply on short sequences, and a small, targeted budget teaches the model to use long positions. The card does not publish long-context benchmark results, so there are no numbers here on how well the model retrieves or reasons at 256K.

What is in the agentic mid-training data?

Mid-training uses curated, higher-quality data with a larger share of agentic traces and tool calls. An agentic trace is a record of a model working through a multi-step task, such as deciding to call a tool, reading the result and continuing. Exposing the base model to these patterns before post-training gives later RL stages something to build on.

The model card adds one detail with practical consequences. Mid-training also included chat data, with and without thinking traces, formatted with the same chat template as LFM3-1.7B. So even the base checkpoint responds to the chat template:

• With apply_chat_template, reasoning is on by default, and responses start with a thinking trace.
• Setting enable_thinking=False prefills an empty thinking trace so the model answers directly.

Liquid AI still recommends LFM3-1.7B, not the base model, for chat, tool calling and agents.

How does post-training get from the base model to LFM3-1.7B?

The card names four post-training methods applied to this checkpoint:

1. Supervised fine-tuning (SFT)
2. Mixed reinforcement learning
3. Multi-domain on-policy distillation
4. Agentic reinforcement learning

It does not state the order, the datasets or the reward designs, and it publishes no benchmark comparisons between base and post-trained models. The final stage, agentic RL, fits with the mid-training data. Tool-call traces in the base give the RL stage a starting behavior worth reinforcing.

Should you use the base model or LFM3-1.7B?

Use the base model when you plan to change the model substantially. The card lists language-specific or domain-specific assistants, training on proprietary data, and experiments with new post-training approaches. For everyday chat, tool calling and agents, use LFM3-1.7B.

Liquid AI points to TRL for both continued pre-training and full post-training. It provides two example notebooks, one for SFT that extracts structured JSON from messy documents, and one for continued pre-training that teaches the model Turkish from raw text. Both start from LFM3-1.7B, and you switch to the base model by setting model_id = "LiquidAI/LFM3-1.7B-Base".

A minimal load looks like this:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "LiquidAI/LFM3-1.7B-Base"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype="bfloat16",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)