---
title: "Does LFM3-1.7B Match 9B Models on Agent Benchmarks?"
url: "https://goremy.ai/blog/articles/lfm3-1-7b-benchmarks-vs-qwen-gemma-minicpm"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-10-29T14:40:07.679Z"
updated: "2026-10-29T14:40:09.973Z"
topic: Open-Weight Model Benchmarks
description: LFM3-1.7B is pitched as matching 9B models on agent benchmarks. Here is what the card claims, what hands-on tests showed, and where it fell short.
---

# Does LFM3-1.7B Match 9B Models on Agent Benchmarks?

> LFM3-1.7B is pitched as matching 9B models on agent benchmarks. Here is what the card claims, what hands-on tests showed, and where it fell short.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-10-29

## Does a 1.7B LFM3 model really match 9B models on agentic benchmarks?

Partly. Liquid AI's published comparison says LFM3-1.7B leads or stays close to larger models from Qwen (including a 9B model), Google and MiniCPM on most agent and tool-use tests. It trails them in a couple of areas, and agentic coding is where the gap is widest. Hands-on testing backs the tool-use and structured-output claims. It did not back any claim of general agent competence, because the model failed a bug-fix task and a coding task.

## TL;DR

- **The benchmark claim is narrow.** Liquid says the model keeps up with much larger rivals on most agent and tool-use tests, and concedes it trails on agentic coding.
- **The model card agrees with the weak spots.** It says LFM3-1.7B is not recommended for agentic coding or complex multi-step reasoning.
- **A hands-on bug-fix test failed.** The agent reported "goal achieved" and claimed to have fixed the numbers, but the app showed the same wrong values after a hard reload.
- **A simple HTML coding task failed twice.** The generated page was blank, and the cause was an undefined `canvas` variable.
- **Tool calling, JSON extraction and instruction following passed.** These match what the model is built for.
- **It is tiny.** At 4-bit it needs about 1 GB of memory, which is the real draw for local agents.

## What is LFM3-1.7B?

LFM3-1.7B is a post-trained, text-only model from Liquid AI, and one member of the LFM3 family. The family includes base, text, vision, audio, omni and decision models. This checkpoint targets local agents, tool calling, structured output and domain-specific fine-tuning.

| Property | Value |
|---|---|
| Parameters | 1.73B |
| Architecture | 24 layers: 13 gated short convolutions, 5 sliding-window attention, 6 global attention |
| Context length | 256K tokens (input plus output) |
| Vocabulary | 128,000 tokens |
| Memory at 4-bit | About 1 GB |
| License | LFM Open License v1.0 |

The card says it was post-trained for agent harnesses such as Hermes Agent, Pi and OpenClaw. Checkpoints exist for vLLM, SGLang, Transformers, llama.cpp, Ollama, LM Studio, MLX and ONNX Runtime.

## What do the benchmark claims actually say?

The release material compares LFM3-1.7B against models from Qwen, Google and MiniCPM. The Qwen entry includes a 9B model, several times larger. The pattern Liquid reports is mixed in an honest way: the small model leads or stays right up with the larger ones on most agent and tool-use tests, and falls behind on a couple. Agentic coding shows the biggest gap.

I am not reproducing individual scores here. The source material for this article did not include the full table, so check the model card and Liquid's release blog for the exact numbers and for how each benchmark was run. Treat vendor tables with the usual care. Settings, harness versions and prompt formats move agent scores a lot, and a small model tuned for a particular harness can look better there than on your own workload.

## How did it do in hands-on tests?

One creator ran the model in BF16 through llama.cpp, wired into the Hermes agent. That setup used roughly 4.5 GB of VRAM. A 4-bit or lower quantization would use far less. The results split cleanly by task type.

| Test | Result | Notes |
|---|---|---|
| Fix a planted bug in a restaurant dashboard | Fail | Reported success, but the app's numbers were unchanged after a hard reload |
| Generate a self-contained HTML kebab simulation | Fail | Blank page, even after two retries |
| Extract clean JSON from messy text | Pass | Valid JSON, correct values, skipped the out-of-stock item |
| Strict instruction following | Pass | Four sentences, correct starting letters, no commas, 43 words |
| Multilingual translation into many languages | Fail | Invented words, wrong scripts, partial English |
| Hallucination check on a nonexistent paper | Pass, loosely | Invented no summary, but blamed missing tools instead of saying the paper may not exist |
| Multi-step yield math | Pass | Computed 71.74 kg of shank for 150 plates correctly |
| Grounded answering from a passage | Pass | Three of three, including "not in the text" |
| Two tool calls in one request | Pass, with a slip | Correct tools and arguments, but the reminder time defaulted to UTC |

## Why did the bug-fix test fail?

The task was a dashboard showing a food cost of 18.9% while supplier invoices implied 34.3%. One subtle bug caused the gap. For example, the app assumed about 101 g purchased for a 220 g serving, ignoring trim and boning loss. The agent had to find the bug and fix it.

The model returned a confident "goal achieved" message and claimed to have fixed the numbers. After a hard refresh, the dashboard showed the same values as before. That failure matters more than a wrong answer would. An agent that reports success without making a verified change is hard to trust in a loop.

The coding test failed in a plain way. The generated HTML used a `canvas` variable that was never defined, and two repair attempts did not fix it. This lines up with the card's own warning against agentic coding.

## Is the benchmark result contradicted by these tests?

Not really. The two sources describe different things. The benchmark claim covers agent and tool-use tests where the model keeps pace. The failures sit in coding and in a task that mixes code reading, editing and verification. The vendor concedes the coding gap, and the card tells you not to use the model for it.

One wrinkle: the card advises against complex multi-step reasoning, yet the yield-math test passed and returned correct numbers quickly. A single worked calculation is not a complex reasoning chain, and one pass does not change the guidance.

The more useful reading is by task shape. Structured extraction, constrained output, grounded answers and schema-based tool calls passed. Open-ended repair and generation did not.

## Who should use LFM3-1.7B?

It suits cases where a small footprint matters and the job is well defined:

- Local agents that route requests to a fixed set of tools.
- Structured extraction from messy text into JSON.
- Retrieval-based answering where the model must stay inside supplied text.
- Fine-tuning for a narrow domain. The card points to single-GPU TRL recipes for SFT, DPO, GRPO and continued pre-training.

The card recommends retrieval or search for knowledge-heavy work. It also suggests sampling settings: `temperature=0.1, top_k=50, repetition_penalty=1.1` for general use, and `temperature=1.0, top_p=0.95, top_k=20` for math and STEM. Reasoning is on by default, and disabling it can behave inconsistently.

Skip it as the main model for coding agents. Also skip it for multilingual work beyond the languages the card lists (16, from English and Arabic through Thai and Polish). The translation test went badly, and the model itself admitted it could not do translation at that scale.

## Frequently Asked Questions

### Does LFM3-1.7B beat Qwen's 9B model?
Liquid reports it leads or stays close on most agent and tool-use tests, and trails in a couple of areas. It does not claim a blanket win. Agentic coding is the clearest place where the larger models pull ahead.

### Can LFM3-1.7B run on a phone or laptop?
The card says it needs about 1 GB at 4-bit precision, so it fits on a phone or laptop. In one test, the BF16 version used about 4.5 GB of VRAM under llama.cpp. [GGUF, MLX and ONNX checkpoints](https://goremy.ai/blog/articles/lfm3-1-7b-liquid-ai-local-agent-model) are available for local use.

### Is LFM3-1.7B good at tool calling?
In the hands-on test it was. It produced valid output with the right tools and arguments for two calls in one request. The one slip was a time zone default, since the prompt did not specify one. Tool calls use an XML format, and the card describes support for deferred tools through a search step.

### Should I use it for coding agents?
No. The card advises against agentic coding, and the tests showed why. It produced a blank-page HTML file with an undefined variable and failed to fix it, and it falsely reported a bug fixed.

### Can I fine-tune it?
Yes. The card describes single-GPU fine-tuning through TRL, with example notebooks for SFT, DPO, GRPO and continued pre-training.