The case for hosting LLMs Locally

The case for hosting LLMs Locally

In an era of increasingly complex and exceedingly capable LLMs being released every day, it’s easy to buy into the marketing hype and be tempted to sign up for the latest and greatest model with all the bells and whistles. The expectation is that these LLMs, like Gemini, Claude, ChatGPT etc. will aid us in increasing our productivity, help us offload the lesser value-add tasks like writing business emails etc.

NOTE: If you are already convinced about hosting LLMs locally and are here to learn about how to do it and best practices around it, you can skip to that step.

Having said that, LLMs tend to show their weakness once we start pushing them beyond simple tasks. They can draft an excellent, concise email, summarize bullet points for a 100-page financial report etc. However, when tasked with doing creative writing, solving an engineering problem, making stock market predictions etc. LLMs tend to give very accurate sounding, well-crafted responses, but when you dig deeper, the answers are often peppered with inconsistencies and hallucinations and outright falsehoods. It is this flaw in their functionality that requires users to be very mindful and diligent in checking these answers. This requires that there will be multiple iterations of refinement, which requires a longer “Context length” and a higher number of tokens per finished task.

Costs:

The above paragraph is a roundabout way of establishing the fact that token costs are not as trivial as the AI companies want us to think. Any task that requires moderate to heavy reasoning, multiple iterations, and longer outputs (A 1000-word essay for example) will make us hit the upper limits of usage, even with paid plans. And we haven’t even discussed non-textual output, such as image modification, generation, analysis etc. These are surprisingly expensive operations and using them at a professional level would cost a fair amount of money. Of course, a model’s capability cannot be measured by tokens alone, as frontier models with more training would perform much more complex tasks, but they do consume a lot more tokens, whereas the cheaper ones cost less, but may not be able to perform certain tasks.

Source: MindStudio.ai

Mind you, we are currently in a phase where every AI lab is competing for market share and is heavily subsidizing the cost of tokens. Especially with flat rate plans, every provider is losing money at a ratio of 1:10 on power users. Even then, the costs are still astronomical for end users. Talk about a lose-lose situation.

The real cost of cloud LLMs isn’t just the single call—it is the exponential token growth caused by retry loops, system prompt context stuffing, and retrieval-augmented generation (RAG).

To be perfectly honest, at this point in time, the costs of LLMs, even at higher utilization rates are high, but not prohibitively high. So, many individuals and companies may balk at the idea of going away from the cloud offerings to hosting their own. While current cost implications do not make a compelling enough case for local LLMs, the next two items, Control and Privacy will absolutely do.

Control:

Model Deprecation and API instability.

Using LLMs on the cloud is always at the mercy of the providers. The inference models are always a Blackbox, and the compute power allocated to us is also an unknown. Companies retire models at will, and change their APIs often with very little notice. While these changes may not cause any major issues to “chat” only users, they can cause havoc to enterprise. If we build our workflows on the assumption that a certain model will behave in such a way, and cost a certain amount, a sudden change to any of those assumptions will lead to wide disruptions. This has happened many times over the past few years much to the chagrin of many developers and product owners.

A prominent example of this scenario was when OpenAI discontinued gpt-3.5-turbo-0301 and code-davinci-002 with little or no notice rendering thousands of workflows unusable and forcing companies to develop fixes urgently.

Privacy and Data Security:

I saved Privacy to last, because it is often ignored by many users. In the pursuit of getting an answer quickly, they provide a lot of very personal, sensitive and financially secret documents to LLMs. Unfortunately, every document you upload, every photo or video you share with an LLM, will potentially become part of its training data. While enterprise contracts restrict training on input or prompted data, it is usually not guaranteed. Hosting the models locally on the other hand ensures that safety.

Given how the training as well as inference is a black box, we have no way of knowing how this information will create security issues for us. Sharing business plans, market research etc. with an LLM would guarantee that your competition will be able to extract that information with a series of well-crafted prompts from the same LLM.

Numerous documented attack vectors exist demonstrating how malicious actors can extract proprietary data from cloud models. While a deep dive into LLM security vulnerabilities is beyond the scope of this article, local deployment eliminates these external risks entirely.

If you are convinced now it’s worth considering self-hosting your own AI, read on.

The three major foundational components of locally hosted LLMs are as follows.

Hardware:

The single most critical—and expensive—component for running local LLMs is the GPU, specifically its Video RAM (VRAM) and memory bandwidth. Because inference relies on keeping model weights active in fast memory, VRAM capacity directly dictates which models you can run. For smaller, highly capable 8B-parameter models (such as Llama 3 or Gemma 2), a GPU with at least 16GB of VRAM—such as an Nvidia RTX 4070 Ti Super or RTX 5070 Ti—is the entry point. Stepping up to larger 70B models requires 24GB or more (e.g., RTX 3090/4090 or multi-GPU configurations). While a solid multi-core CPU and 32GB to 64GB of system RAM are necessary to support the system, the graphics card remains the primary engine driving local AI performance.

A typical LLM rig setup can be seen below.

Source: promptsquorum.com

This below is my personal setup. This was meant only as a proof-of-concept, and is unlikely to be useful beyond the simple chat engine and simple automations.

Compute vs. Memory: While compute power matters, VRAM capacity and bandwidth—not raw GPU speed—are the primary bottlenecks for running inference.

RAM & CPU Requirements: System RAM (64 GB) is secondary unless offloading weights from the GPU. Additionally, Intel i9 CPUs are not a hard requirement; modern mid-range CPUs (or Apple Silicon Unified Memory) handle local inference effortlessly because the GPU handles the heavy lifting.

GPU & VRAM Realities: 16 GB VRAM is great for 8B-parameter models (like Llama 3.1 8B or Gemma 2 9B at 4-bit or 8-bit quantization), but running larger models (e.g., 70B parameters) locally requires 24 GB+ (like an RTX 3090, 4090, or 5070 Ti) or dual-GPU setups.

Software (LLM Host):

When running models locally, two primary platforms dominate the ecosystem:

• Ollama (CLI-First): Preferred by developers, Ollama operates as a lightweight terminal-based tool and background service. It simplifies downloading, configuring, and serving models via a command line interface. Because it runs without a graphical interface, background overhead is minimal, maximizing available resources for model inference.

Open source LLM models can be downloaded from public repositories like huggingcace (literally hundreds to choose from)

Note that “Abliterated” models are specialised models where the safety guardrails have been removed from these very powerful models. These models provide answers to pretty much any question, no matter their legality. These models are used by some sections of society who resent censorship of any sort. Exercise caution.

We can prompt our chosen model directly from the command line as below. ( Proper chat-style interface will be shown later)

• LM Studio (GUI-First): Designed for users who prefer a graphical interface, LM Studio offers an intuitive visual workspace. It features a built-in Hugging Face model browser, real-time token performance monitoring, and parameter controls (such as context window and temperature adjustments). While the GUI introduces slight desktop memory overhead, inference performance remains equivalent to Ollama.

Both applications feature built-in OpenAI-compatible API servers, allowing seamless integration with local developer tools, scripts, and third-party extensions. Choosing between them depends on whether you prefer terminal efficiency or visual control.

Once hosting has been setup, the next step is to provide a browser-based user interface which is similar in look and feel to the commercial LLMs. This can be achieved by using the open-source tool Open Web UI

Once installed, the LLMs we spun up earlier can be accessed from a browser.

Open WebUI is an extensible, self-hosted web interface designed to bring a slick, ChatGPT-style chat experience to self-hosted LLMs. While inference engines like Ollama or LM Studio handle the underlying compute and API endpoints, Open WebUI sits on top to provide the visual interface users actually interact with—complete with user management, custom system prompts, document uploads (RAG), web search integration, and voice support.

Without a frontend like Open WebUI, self-hosting often requires interacting through a bare terminal or restricted local app windows; Open WebUI bridges that gap by delivering a feature-complete, multi-user workspace that makes running local AI models feel just as polished and accessible as top commercial cloud services.

This lays the foundation for self-hosting your own AI. With this setup, you have a fully functional local LLM paired with a familiar chat interface—delivering most of the utility of cloud-hosted alternatives with total privacy and control.

However, a chat interface is only the beginning. The true power of LLMs lies in agentic workflows and local RAG environments, moving beyond basic Q&A into true automation. Part 2 of this series will cover setting up local RAG and building autonomous agents.

If you found this guide helpful, please share it on LinkedIn and subscribe to my tech channel on YouTube!