Building a Local AI Agent with a Raspberry Pi and Read-Only Postgres

The short answer

Yes, you can run an LLM locally on a Raspberry Pi 5, and you can wire it to a real database safely. The catch is model size, not feasibility. Stick to models under 4 billion parameters and you get usable, private, offline inference for well under $150 in hardware.¹²

I wanted to test a specific claim that gets thrown around in Shadow AI debates: that you need cloud GPUs and a managed database to build anything useful. So I built the smallest possible version of an internal AI agent. A Raspberry Pi 5 running a local model through Ollama, talking to a Postgres database through a read-only role, answering questions about data it could never modify. If this cheap setup can hold a governance line that some company AI tools can't, that says something about how low the bar for "tightly governed" actually is.

The hardware reality check

A Raspberry Pi 5 launched at $60 for 4GB and $80 for 8GB.³ Those prices did not last. An AI-driven memory shortage has pushed DRAM costs up sharply, and Raspberry Pi's own CEO confirmed a second price increase in three months, with the 8GB board now around $125 and the 16GB board at $205, over 70% above its original MSRP.⁴ Even at the higher price, an 8GB Pi 5 is still a fraction of what a cloud GPU instance costs per month, and you own it outright once you pay for it.

That ownership point matters more than the sticker price. A Pi you buy is a Pi you keep. A cloud LLM endpoint is a bill that renews forever, which is the same math this publication has run on renting tokens versus owning quantized models.

What actually runs, and what falls over

The Pi 5 has a quad-core Cortex-A76 CPU and no dedicated GPU for inference, so everything runs on CPU. That sets a hard ceiling.

Independent testers landed on a similar pattern. Models in the 2B to 4B parameter range run well. Anything at 7B parameters either crawls or refuses to load. In one hands-on test of nine models, Gemma2 2B and Qwen2.5 3B rated the best experience, using 3 to 5.4GB of RAM with fast, coherent responses, while Llama 2 7B and Codellama 7B simply would not run on an 8GB board.¹ Mistral 7B did run, but inference took around six minutes for a single task.¹

A more rigorous benchmark from a security research lab measured actual tokens per second rather than vibes. Small models like gemma3:1b and qwen2.5:1.5b delivered the best combination of speed and low memory use. Qwen2.5 3B scored highest on task accuracy but generated at roughly 5 tokens per second, slow enough that a 5,000-token prompt took about 15 minutes to process.² The same lab found llama.cpp ran 10% to 20% faster than Ollama on the Pi 5, mostly because Ollama's default 4096-token context window and lack of thread control waste headroom on constrained hardware.²

The honest takeaway: this is not a chatbot replacement. It is a narrow-task machine. Good for classification, summarizing a short document, answering a bounded question against a small dataset. Bad for anything that wants long context or fast turnaround.

Tokens per second by model size, Pi 5

Speed drops fast as parameter count climbs, and Qwen2.5 3B's accuracy lead costs real latency.

The read-only Postgres piece

The interesting part of this experiment was not the model. It was the database wiring. If an AI agent can read your data but never touch a write path, most of the scary failure modes in Shadow AI conversations disappear before they start.

Postgres makes this cheap to enforce. Create a role, grant it USAGE on the schema and SELECT on the tables, and it cannot INSERT, UPDATE, or DELETE no matter what the model tells it to do.⁵

CREATE ROLE agent_reader LOGIN;
GRANT USAGE ON SCHEMA app TO agent_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA app TO agent_reader;
ALTER DEFAULT PRIVILEGES IN SCHEMA app GRANT SELECT ON TABLES TO agent_reader;

That last line matters. Grants made today don't automatically apply to tables created tomorrow, so without ALTER DEFAULT PRIVILEGES, a new table silently becomes invisible or, worse, someone re-grants broad access out of frustration and blows the whole model open.⁵ Postgres 14 added the pg_read_all_data predefined role specifically to simplify this, letting you grant read-only access across every object regardless of who created it.⁵

Run that role behind a Pi, and the worst a misfiring model can do is return a wrong answer. It cannot write a wrong answer into your production data. That is a governance boundary you can actually verify by trying to break it, unlike a vendor's privacy policy.

What this proves and what it doesn't

This setup proves you can build a small, auditable, offline internal tool for close to nothing: a $95 to $125 board, a quantized model, and a database role that makes destructive actions structurally impossible rather than merely policy-forbidden. For a narrow job like "summarize last week's support tickets" or "flag anomalies in this one table," that is a complete, working system.

It does not prove the Pi replaces a real inference server for anything with real latency requirements or long context. And it does not remove the need for someone to actually design the read-only boundary. The database security is good specifically because someone configured it correctly. A model with unrestricted write access on beefier hardware is not more governed just because the box costs more.

The bigger point is about defaults. Most teams reach for a hosted API and a shared database credential because it's the fastest path to a demo. The Pi-and-read-only-Postgres version takes longer to set up and is slower to answer. But it is a tool your team owns outright, with a permission boundary you can read in four lines of SQL instead of trusting to a settings page. That's the same argument this publication makes about orchestrating your own office of clones instead of renting agent platforms wholesale: cheap hardware plus tight permissions beats expensive infrastructure plus implicit trust. If you're weighing whether to formalize this kind of homegrown setup into something a whole team can rely on, tools like Remy exist for exactly that governance gap between a one-off Pi project and a fully rented SaaS stack.

FAQ

Can a Raspberry Pi actually run an LLM, or is this just a toy? It genuinely runs models, but only in the 1B to 4B parameter range with acceptable speed. Testing on a Pi 5 8GB found small models like Gemma2 2B and Qwen2.5 3B responsive and accurate, while 7B models were either unusably slow or failed to load at all.¹²

Which inference engine should I use, Ollama or llama.cpp? Ollama is easier to set up and manage. llama.cpp runs 10% to 20% faster on the Pi 5 and gives you direct control over context length and CPU thread allocation, which matters more on constrained hardware than on a desktop.²

How do I stop the AI agent from writing to my database by accident? Create a dedicated Postgres role with only SELECT and USAGE grants, and set ALTER DEFAULT PRIVILEGES so new tables inherit the same restriction. On Postgres 14+, the built-in pg_read_all_data role does this in one grant.⁵

Is a Raspberry Pi actually cheaper than cloud inference? For a narrow, low-volume internal tool, yes, once you own the hardware. But Pi 5 prices have risen sharply due to a memory shortage, with the 8GB board up to roughly $125 and the 16GB board at $205, so the up-front math is worse than it was two years ago.⁴

What tasks is this setup actually good for? Bounded, low-context jobs: summarizing a short document, classifying a small batch of records, answering yes/no questions against a limited dataset. It struggles with long prompts and multi-turn conversations because token generation slows to a handful per second on larger models.²