Ollama Linux: Run LLMs Locally in 10 Minutes (2026)

DIRECT ANSWER

Ollama allows you to run Large Language Models (LLMs) like Llama 3 and Mistral directly on your Linux server, ensuring complete data control and reducing costs, with a setup that takes less than 10 minutes to get operational.

Quick Snippet: Installation and First Run

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull Llama 3
ollama pull llama3

# Run Llama 3 and start chatting
ollama run llama3

Narrative Introduction

I’ve witnessed too many scenarios where the rush to adopt Artificial Intelligence solutions led to severe compromises in data security. I recall a client, a public healthcare organization with thousands of endpoints, sending sensitive data via API to a third-party LLM. Every single query, even the most innocuous, potentially exposed confidential information to the external provider. The solution? Run the LLM locally. With Ollama, we deployed Llama 3 on a dedicated Linux server with a GPU, achieving the same response quality with zero data leakage. The entire setup was operational in under 10 minutes. This article will guide you step-by-step to replicate the same setup, ensuring privacy, control, and reduced costs for your AI needs.

Prerequisites / Test Environment

To follow this guide, you will need a Linux system (Ubuntu, Debian, Fedora, Arch Linux, or derivatives) with internet access. Hardware with at least 8 GB of RAM is recommended for smaller models (like Mistral), and ideally a GPU with updated drivers for optimal performance, although Ollama can run in CPU-only mode for testing or very small models. I tested this procedure on an Ubuntu 24.04 LTS VM and a physical server with Debian 12 and an NVIDIA RTX 3060.

Why Run an LLM Locally — Privacy, Costs, Latency

The adoption of LLMs has revolutionized many industries, but using third-party cloud services raises significant concerns, especially for organizations handling sensitive data. Running an LLM locally offers invaluable advantages:

• Data Privacy and Security: This is the primary reason. When an LLM runs on-premise, your data never leaves your infrastructure. There is no risk of data leakage to third parties, which is crucial for compliance with regulations like GDPR or NIS2. A 2024 survey revealed that 68% of companies are concerned about data privacy when using cloud-based LLMs (Source: IBM AI Adoption Index 2024).
• Costs: Pay-as-you-go LLM API usage can become extremely expensive, especially with high workloads. By running a model locally, costs are limited to hardware and energy, eliminating recurring external API fees. This ensures financial predictability.
• Latency: Locally processed queries eliminate network latency. Responses are nearly instantaneous, improving user experience in real-time applications and the efficiency of automated processes.
• Control and Customization: You have complete control over the model, being able to choose specific versions, optimize them for your datasets (fine-tuning), and deeply integrate them with your existing applications without relying on specific third-party APIs.

Installing Ollama on Linux (Ubuntu/Debian/Arch)

Installing Ollama is surprisingly simple and requires a single command line. Ollama handles detecting your operating system and installing the correct binaries, including necessary drivers for supported GPUs (NVIDIA and AMD).

Open a terminal and execute the following command:

curl -fsSL https://ollama.com/install.sh | sh

This script will download and install Ollama on your system. Once completed, Ollama will be available as a terminal command. You can verify the installation and version with ollama --version.

Downloading and Running Your First Model (Llama 3, Mistral, Gemma)

With Ollama installed, the next step is to download an LLM. Ollama offers a wide library of popular models. To start, I recommend Llama 3, one of the most performant and versatile models available.

To download Llama 3 (8B version, optimized for resource-constrained systems but still powerful):

ollama pull llama3

The download may take several minutes depending on your connection and the model size (Llama 3 8B is approximately 4.7 GB). Once downloaded, you can start an interactive chat session with the model:

ollama run llama3

Now you can interact with Llama 3 directly from your terminal. To exit, type /bye or press Ctrl+D.

You can list downloaded models with:

ollama list

To run Ollama as a background service, useful for API integrations:

ollama serve &

Ollama REST API — Integrating with Python and Bash Scripts

One of Ollama’s strengths is its RESTful API, which allows for easy integration of LLMs into custom applications and scripts. The API is compatible with OpenAI specifications, making migration from cloud services to local solutions very smooth.

To make a sample query to Llama 3 via the API, ensure Ollama is running in serve mode (as shown above) and then use curl:

curl http://localhost:11434/api/generate \n  -d '{"model": "llama3", "prompt": "Explain cgroups in Linux in 20 words"}'

This HTTP POST call will send a prompt to the Llama 3 model and receive a JSON response. You can easily integrate this into Python scripts using the requests library or into Bash scripts to automate tasks like report generation, log analysis, or summary creation.

Open WebUI — ChatGPT-like Interface for Ollama

For those who prefer a graphical interface to interact with their local LLMs, Open WebUI is the ideal solution. It is an open-source user interface that connects to Ollama, providing a ChatGPT-like experience directly in your browser. Installation is simple, typically via Docker.

To install Open WebUI (requires Docker installed):

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Once the container is started, you can access Open WebUI from your browser at http://localhost:3000. From there, you can select available Ollama models and start chatting in a user-friendly environment.

Hardware Requirements: How Much RAM Does Each Model Need?

Large Language Models are resource-intensive, especially concerning RAM and GPU VRAM. The amount of memory required depends on the model’s size (expressed in billions of parameters, ‘B’).

| Model (Example) | Parameters | Minimum Recommended RAM/VRAM |

| :—————- | :——– | :————————— |

| TinyLlama | 1.1B | 4GB |

| Mistral / Llama 3 | 7B-8B | 8GB |

| Llama 3 | 70B | 64GB |

An enterprise environment with 300+ VMs and 2,000 endpoints often has servers with adequate resources, but careful planning is essential. If you don’t have a dedicated GPU, Ollama will use the CPU, but performance will be significantly lower. I’ve observed that a server with 32GB of RAM and a modern CPU can handle Llama 3 8B in CPU mode for non-critical tasks, but for intense workloads or low latency, a GPU is indispensable. NVIDIA (with CUDA) is currently the most performant choice, but AMD support is growing. For more details on hardware recommendations, refer to the official Ollama documentation.

Recommended Models for Sysadmins (Code, Log Analysis, Scripting)

For a SysAdmin, LLMs can be powerful tools for automation, analysis, and debugging. Here are some models that have proven particularly useful:

• Llama 3 (8B/70B): Excellent for code generation (Python, Bash, Ansible), analyzing command outputs, and general troubleshooting. The 8B version offers a good balance between performance and hardware requirements.
• Mistral (7B): Excellent for programming tasks and text generation. It is known for being very efficient and performant on limited hardware, ideal for scripts and automation.
• Code Llama (7B/13B/34B): Specifically trained for code generation and understanding. Indispensable for writing complex scripts, refactoring, or debugging.
• Gemma (2B/7B): A lightweight model from Google, very fast and suitable for text generation and summarization tasks, useful for documenting configurations or analyzing small log files.

Running Ollama as a systemd Service

In a production environment, it is crucial for Ollama to run as a system service, managed by systemd. This ensures that Ollama starts automatically at system boot and is monitored and restarted in case of crashes.

The Ollama curl installation script should already configure a systemd service. You can check its status with:

systemctl status ollama

If it is not active or enabled, you can do so manually:

systemctl enable ollama
systemctl start ollama

This will ensure Ollama is always available for your applications and APIs, even after a server reboot. Read also: Hardening SSH en Linux: Guía Completa 2026

Common Errors and Troubleshooting

• Memory Issues: If the model fails to load or crashes, check RAM/VRAM requirements. ollama run usually shows an explicit error if memory is insufficient. You might need to download a smaller model or add more RAM/GPU.
• GPU Drivers: On Linux systems, ensure your GPU drivers are correctly installed and updated. nvidia-smi (for NVIDIA) or rocminfo (for AMD) can help diagnose GPU issues.
• Port Conflicts: If the Ollama REST API (port 11434) is not accessible, verify that no other services are listening on the same port with sudo lsof -i :11434. You might need to restart the Ollama service or change its port, although this is not a direct Ollama configuration option but a system-level one.
• Permissions: The Ollama installation script handles permissions, but in rare cases, issues with permissions on the installation directory or models (~/.ollama) can cause errors. sudo chown -R $USER:$USER ~/.ollama can resolve this in some cases.

FAQ — Frequently Asked Questions

How can I update Ollama to the latest version?

To update Ollama, you can run the installation script again: curl -fsSL https://ollama.com/install.sh | sh. This will download and install the latest available version, preserving your existing models. It’s good practice to periodically check for updates to benefit from the latest optimizations and features.

Can I run multiple models simultaneously with Ollama?

Yes, Ollama supports running multiple models concurrently, provided your hardware has sufficient RAM/VRAM to load them. Each loaded model will consume a portion of the available resources. You can start separate sessions with ollama run and ollama run or interact with different models via the API.

What is the difference between ollama run and ollama serve?

ollama run starts an interactive chat session with a specific model in the current terminal. ollama serve starts the Ollama server in the background, exposing the REST API on port 11434, without starting a chat session. This is the way to make Ollama available for external applications or web interfaces like Open WebUI.

Is it possible to run Ollama on Windows or macOS?

Yes, Ollama also offers native installers for Windows and macOS, allowing you to run LLMs locally on these operating systems. The user experience and features are similar to the Linux version, with the same advantages in terms of privacy and control. You can find downloads directly on the official Ollama website.

Conclusions with Operational Takeaways

Running LLMs locally via Ollama represents a paradigm shift for organizations seeking to leverage Artificial Intelligence without compromising data privacy or facing unpredictable costs. The installation is straightforward, integration is flexible, and control is total. I recommend starting with Llama 3 8B, a robust and versatile model, and exploring API REST integration to automate your workflows. This is not just a financial saving, but an investment in the security and autonomy of your IT infrastructure. Do not let your sensitive data end up in someone else’s cloud when you can have the same computational power in-house, with full control.

Updated: May 2026