The problem I need to solve is concrete: install a truly usable open-model environment on my own local development machine.

Not “run a demo once,” and not “pull the largest model possible to prove the machine is decent.” I want something that can stay in my daily development flow for the long term: coding, summarizing material, light data analysis, local APIs, occasional multimodal tests, while not making the machine unusable for normal work.

This article will keep being updated. The first version records the runtime decisions: whether Ollama should run on Windows or WSL, what faster inference options exist beyond Ollama, and which models this machine is realistically suited for. Later updates will add real installation notes, downloads, VRAM usage, token speed, context length, and task performance.

Current Machine

This is not a dedicated GPU server. It is a daily development machine:

Item	Configuration
Host system	Windows
WSL system	Ubuntu 26.04 LTS
GPU	NVIDIA GeForce RTX 4060 Laptop GPU
VRAM	8GB
Host memory	64GB
WSL memory	32GB, about 31GiB visible inside the system
Main use	Programming, WSL development, local model experiments

The character of this configuration is clear: CPU and memory are enough for daily development. With 32GB assigned to WSL, even if a model cannot fully fit into VRAM, there is still enough system memory as a fallback. But 8GB VRAM means this is not a large-model workstation. Model choices need to be realistic. 4B to 9B is the comfort zone. 12B is testable. Above 14B, you must accept obvious slowdowns, partial offload to system memory, and more limited context.

The system has been upgraded to Ubuntu 26.04 LTS, which is friendlier to local LLM toolchains. A newer base system means newer glibc, Python, compilers, system libraries, and package repositories. Later, when compiling llama.cpp, installing CUDA-related Python packages, or running newer inference services, old-system dependency friction should be lower. Of course, inference speed is ultimately decided by GPU, VRAM, quantization format, and backend. The OS version only reduces environment friction.

Ollama on Windows or WSL

If the goal is only chatting, trying models, or providing a local API to Windows desktop apps, I lean toward native Windows Ollama.

The reason is simple: one fewer environment layer. After installation, the Windows version is a background service, the default API is http://localhost:11434, and Windows clients, browsers, Chatbox, Open WebUI, and editor plugins can access it naturally. The GPU driver is also on the Windows side, so the chain is shorter and troubleshooting is more direct.

But my normal development environment is mainly WSL. Python, Node, Rust, database clients, scripts, benchmarks, and log analysis mostly run in Linux. In that case, putting Ollama in WSL has one advantage: script calls, local file paths, and automated tests all stay in one environment, with less Windows/Linux path translation.

My judgment:

Scenario	Better Choice
Desktop chat, simple experience, Windows client calls	Native Windows Ollama
WSL development, Python scripts, model evaluation, Linux toolchain integration	Ollama inside WSL
Multiple model services at the same time	Keep one main service to avoid VRAM contention

One important warning: do not run both Windows Ollama and WSL Ollama at the same time and have both load models. With only 8GB VRAM, two services fighting for memory quickly becomes “both can start, neither feels good.”

Where Model Files Should Live

If the WSL virtual disk is still on the C drive, model files can fill it quickly. One quantized 8B model is several gigabytes, one 12B model can approach 8GB, and trying a few versions quickly becomes expensive.

I prefer moving the whole WSL instance to the D drive, or using wsl --import to create a new Ubuntu distribution directly on the D drive. Then model caches, project dependencies, Docker images, and vector indexes all live inside the same ext4.vhdx. File I/O is more natural than placing hot model files under /mnt/d/....

This test only uses ~/.ollama/models inside WSL and does not reuse the existing Windows-side Ollama model cache. The benefit is a clear boundary: native Windows Ollama and WSL Ollama do not share model files and do not affect each other through version, permissions, paths, or background services.

If many models will be kept for the long term, the cleanest solution is still to move the WSL distribution itself to a larger disk and keep ~/.ollama/models in the Linux filesystem, instead of directly mounting a hot model directory from the Windows filesystem.

Options Beyond Ollama

Ollama’s advantage is convenience. Its downside is also clear: it has a relatively thick wrapper, and many details are harder to control. When you need to squeeze performance, tune context length, tune GPU offload, or test different quantized files, lower-level tools matter.

llama.cpp

For single-machine local inference, llama.cpp remains the most serious tool to consider. It supports GGUF, has a mature CUDA path, exposes fine-grained parameters, and is suitable for observing VRAM, speed, and context length while testing models.

I will use it later as a performance baseline: run the same GGUF model once through Ollama and once through llama.cpp server, then see how large the gap really is.

LM Studio

LM Studio is better for desktop experience. It has a GUI, can download models, adjust runtime parameters, and expose an OpenAI-compatible API. For Windows users, it is much friendlier than compiling llama.cpp yourself.

If you do not want to study a pile of commands first, LM Studio is a good second choice. It is also useful for quickly judging whether a GGUF model is worth deeper testing.

ExLlamaV2 / TabbyAPI

If using EXL2 or GPTQ quantization, tools such as ExLlamaV2 are often very strong for single-user generation speed. But they are more picky about model format and deployment, and less convenient than GGUF + Ollama / llama.cpp.

I will evaluate them later. First, the main models need to run well; then it is worth asking whether switching format for speed is worth it.

vLLM

vLLM is designed for concurrent, high-throughput API services. The problem is that this machine has only 8GB VRAM, so many scenarios where vLLM shines do not fit it.

On 24GB, 48GB, or multi-GPU machines, vLLM deserves serious investment. On this development machine, it is not the first priority.

A simple ordering:

Goal	Recommendation
Fastest way to start	Ollama
Windows desktop experience	LM Studio
Single-machine performance and control	llama.cpp
EXL2/GPTQ high-speed inference	ExLlamaV2 / TabbyAPI
Concurrent service	vLLM

Which Models Fit This Machine

The reality of 8GB VRAM is simple: do not start with 30B or 70B. Being able to run and being pleasant to use are different things.

I would test models in this order:

Priority	Model	Use	Expectation
P0	Qwen3 8B	Chinese, code, daily Q&A	Most likely to become the main model
P0	Qwen3 4B	Light tasks, summaries, classification, script help	Fast, suitable for staying resident
P1	Gemma 4 E4B	Lightweight multimodal experiments	Used to verify image, audio, and video-frame workflows
P1	Gemma 4 12B	Higher-quality multimodal and complex tasks	Can run, but may be slow
P1	DeepSeek-R1-Distill-Qwen 7B/8B	Reasoning tasks	Good for targeted tests, not necessarily default chat
P2	Qwen3 14B	Stronger Chinese and reasoning	Only worth low-quantization experiments; may not feel smooth
P2	Llama 3.1/3.2 8B	English general use and ecosystem comparison	Stable, but Chinese may not be best

If installing only one, I would start with Qwen3 8B. It is the most likely model to balance Chinese, code, and general Q&A.

If keeping a lightweight tool model as well, I would add Qwen3 4B. Such models are not necessarily the smartest, but daily summarization, rewriting, and simple code assistance care more about response speed.

If the goal is multimodal, I would start with Gemma 4 E4B, not jump directly to gemma4:12b. The 12B model is more attractive, but 8GB VRAM makes it less nimble in many scenarios.

First-stage Installation

The first stage does not try to install everything. It only builds a minimal loop that can support judgment. These are notes from the real process, not a polished “ideal command list.”

First, confirm the environment:

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
cat /etc/os-release
free -h

Inside WSL, this machine can see the RTX 4060 Laptop GPU with 8188 MiB VRAM and NVIDIA driver 572.83. The WSL system is Ubuntu 26.04 LTS, visible memory is about 31GiB, and swap is 8GiB.

The official Ollama Linux install script writes to /usr/local/bin and creates a systemd service, which requires root privileges. To get the experiment running first, I did not modify system directories; I installed it into the current user’s directory:

mkdir -p ~/.cache/local-llm ~/.local/ollama ~/.local/bin ~/.ollama/logs
wget -c --progress=dot:giga \
  https://github.com/ollama/ollama/releases/download/v0.30.10/ollama-linux-amd64.tar.zst \
  -O ~/.cache/local-llm/ollama-linux-amd64-v0.30.10.tar.zst
tar --zstd -xf ~/.cache/local-llm/ollama-linux-amd64-v0.30.10.tar.zst -C ~/.local/ollama
ln -sf ~/.local/ollama/bin/ollama ~/.local/bin/ollama
ollama --version

A practical pitfall: newer Ollama Linux packages are .tar.zst. If you still manually download the old .tgz path, you will hit a 404. The official install script handles this automatically, but manual installation requires checking the release asset yourself.

This installation used ollama version is 0.30.10. The package was about 1.3GB, downloaded from GitHub releases, at roughly 1.6MB/s, taking 13 minutes and 35 seconds. Later model files are larger, so network quality directly affects the installation experience.

Then start the Ollama service as the current user:

setsid env \
  PATH="$HOME/.local/ollama/bin:$PATH" \
  OLLAMA_HOST=127.0.0.1:11434 \
  OLLAMA_MODELS="$HOME/.ollama/models" \
  OLLAMA_NO_CLOUD=true \
  OLLAMA_DEBUG=INFO \
  ~/.local/bin/ollama serve > ~/.ollama/logs/ollama-serve.log 2>&1 < /dev/null &

After the service starts, curl http://127.0.0.1:11434/api/tags confirms the API is reachable. The log shows two inference backends:

CUDA0 NVIDIA GeForce RTX 4060 Laptop GPU, total 8.0 GiB, available 6.9 GiB
Vulkan0 Microsoft Direct3D12 (NVIDIA GeForce RTX 4060 Laptop GPU), total 7.8 GiB, available 7.0 GiB

This means Ollama can already use the NVIDIA GPU inside WSL. Installing CUDA Toolkit is not required for Ollama itself. nvcc affects building CUDA llama.cpp, not Ollama inference.

For llama.cpp, I first built the CPU version:

mkdir -p ~/opt
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git ~/opt/llama.cpp
cmake -S ~/opt/llama.cpp -B ~/opt/llama.cpp/build-cpu -DCMAKE_BUILD_TYPE=Release
cmake --build ~/opt/llama.cpp/build-cpu --config Release -j"$(nproc)"

The first full clone was interrupted under slow network conditions. --depth 1 worked. The build used gcc 15.2.0 and cmake 4.2.3; generated binaries are under:

~/opt/llama.cpp/build-cpu/bin/

The build log showed that Qwen3 and Gemma4 model architectures were included, but this is only the CPU version. For a fair performance comparison with llama.cpp, a CUDA build or a suitable prebuilt CUDA package is still needed.

The initial Ollama pull plan was:

ollama pull qwen3:4b
ollama pull gemma4:e4b
ollama pull qwen3:8b
ollama pull gemma4:12b

The qwen3:4b model blob is about 2.5GB. At around 2MB/s, the download is workable but not fast. I did not reuse any old Ollama cache on the Windows side. Models inside WSL live only under ~/.ollama/models, avoiding interference between native Windows Ollama and WSL Ollama.

Qwen3 4B Quick Test

qwen3:4b finished downloading:

NAME        ID              SIZE
qwen3:4b    359d7dd4bcda    2.5 GB

On the first call, Ollama needed to load the model into GPU memory, with a cold start of about 12.3 seconds. After warm-up, later calls loaded in 0.13 to 0.16 seconds. With a Chinese short task around 120 to 160 tokens, generation speed was about 38 to 43 tokens/s, and VRAM usage was about 2.7GB to 2.8GB.

These numbers show that Qwen3 4B is easy for this RTX 4060 Laptop 8GB machine. It is suitable as a resident model for summaries, classification, short rewrites, and lightweight code explanation.

But it also revealed a real issue: the current Ollama template for qwen3:4b automatically inserts <think> in the final assistant turn. Default API responses may put content into the thinking field, or still mix reasoning traces into the body after think:false. In other words, the model runs and speed is good, but if it is to be wired directly into daily tools, the “do not output thinking” template issue needs to be handled, or a more suitable instruct/no-thinking version should be used.

This cannot be ignored. Many local models look great in token/s tests, but once connected to real applications, output format, system prompts, templates, and stop words become the actual experience.

First llama.cpp Comparison

The large blob downloaded by Ollama begins with standard GGUF:

4747 5546 ... GGUF

So it can be read directly by llama.cpp. The current build is CPU-only, not CUDA, so this round is only a “CPU baseline,” not a fair performance comparison.

llama-bench result:

qwen3 4B Q4_K - Medium, CPU, 16 threads
prompt processing: 4.54 tokens/s
text generation:   12.42 tokens/s

Compared with Ollama through GPU at 38 to 43 tokens/s, CPU mode is clearly not suitable as the daily path. Still, the result is useful: the same model file can be read by llama.cpp, so once CUDA llama.cpp is added later, a fairer backend comparison becomes possible.

Gemma 4 E4B Download

The gemma4:e4b manifest could be fetched normally, but the model file was not a tiny package. It was 9.6GB. At around 2MB/s, the full download, verification, and manifest write took more than an hour.

This is another reminder: multimodal models cannot be judged only by the parameter count in their name. Vision, audio, projection layers, tokenizer, and quantization all affect final download size. For an 8GB VRAM development machine, Gemma 4 E4B is still worth testing, but it is no longer a “small model pulled casually.”

After completion, the model list was:

NAME          ID              SIZE
gemma4:e4b    c6eb396dbd59    9.6 GB

Text cold load took about 14.4 seconds; after warm-up, load dropped to about 0.7 seconds. Short-text generation speed was about 54 to 56 tokens/s, and warm VRAM usage was about 3.1GB.

Gemma 4 E4B also has an output-format pitfall in Ollama: default generate calls produce tokens, but Chinese answers are often filtered into an empty response. With raw: true, plain Chinese text works:

本地多模态模型非常适合进行离线、隐私敏感的图像识别和语音转录等基础处理。

Image input cannot be used together with raw: true, or it reports Failed to tokenize prompt. Switching to /api/chat and adding think:false makes image understanding return normal Chinese text. When testing with this article’s cover image, it identified themes such as “developer, multi-screen, AI, CUDA, high-performance computing,” showing that the multimodal path works. Without think:false, content went into the thinking field and the body was empty.

So the first-stage conclusion for Gemma 4 E4B is: the model runs, multimodal works, and speed is good, but integration must explicitly handle think / parser / raw mode, or it is easy to get an empty body.

Qwen3 8B Quick Test

qwen3:8b is a 5.2GB download:

NAME        ID              SIZE
qwen3:8b    500a1f067a9f    5.2 GB

The first cold load took about 16.8 seconds, and warm load was about 0.2 seconds. Default calls still put content into thinking instead of response; with think:false, Chinese answers returned normally.

Measured on the same short Chinese task:

Model	Mode	Cold/Warm	Generation Speed	Behavior
Qwen3 4B	default	warm	about 42 tok/s	Output easily goes into thinking
Qwen3 4B	think:false	warm	about 43 tok/s	Reasoning traces may still appear; template needs more work
Qwen3 8B	default	cold	about 21.9 tok/s	response empty, long thinking
Qwen3 8B	think:false	warm	about 25.7 tok/s	Chinese body normal

After warm-up, qwen3:8b used about 4.9GB VRAM. For an 8GB VRAM machine, this is still comfortable. But if a browser, IDE, and multiple model services are open at the same time, VRAM cleanup matters.

From the first round, Qwen3 8B is more suitable as the daily main model, while Qwen3 4B is better for resident light tasks. Their shared issue is that local integration must verify the Ollama template, think parameter, and returned fields, not just model name and speed.

Gemma 4 12B Download

gemma4:12b finished downloading and registered as 7.6GB:

NAME          ID              SIZE
gemma4:12b    4eb23ef187e2    7.6 GB

It is smaller than gemma4:e4b, but runtime is obviously heavier. Text cold load took about 43 seconds. Default mode still generated tokens but returned an empty body. raw: true returned Chinese text, but included channel markers such as <|channel>thought, which need cleanup at the application layer or further template adjustment.

After warm-up, text generation speed was about 8.8 tokens/s. Image input through /api/chat plus think:false returned normal Chinese, at about 8.7 tokens/s. Image-understanding quality was good: it identified WSL, GPU monitoring, CUDA, AI Pipeline, and technical tutorial scenes. But 40-second-level latency for an image answer is not suitable for a daily lightweight assistant.

For VRAM, Gemma 4 12B used about 5.8GB when warm in text mode. It can run, but on an 8GB development machine, 12B is better for targeted testing and higher-quality multimodal exploration, not for staying resident.

First Installation Result

At this point, all four planned models were installed under WSL ~/.ollama/models:

NAME          SIZE
qwen3:4b      2.5 GB
qwen3:8b      5.2 GB
gemma4:e4b    9.6 GB
gemma4:12b    7.6 GB

Including Ollama and model cache, ~/.ollama/models is about 24GB. This number is much larger than parameter count alone suggests. If more models will be tested later, disk planning needs to happen early.

The first round is rough, but several judgments are already clear:

Model	Suitability	First Impression
Qwen3 4B	Good resident light-task model	Fast, but thinking/template handling is needed
Qwen3 8B	Best daily main model	Better balance of quality and speed; usable with think:false
Gemma 4 E4B	Good for multimodal experiments	Image path works and speed is good, but parser may return empty body by default
Gemma 4 12B	Good for targeted tests	Runs and image understanding is steadier, but clearly slow

llama.cpp currently has a CPU build and has run llama-bench with the same GGUF blob from Qwen3 4B. CPU generation was about 12.4 tokens/s, far below Ollama with GPU at 38 to 43 tokens/s. The next step is CUDA llama.cpp; otherwise it is not a fair comparison.

Later Test Plan

This article only gets the installation path running. Simple tests first use qwen3:4b for Chinese Q&A, code, summaries, and structured output. The second article will compare the models in more detail, including Qwen3 4B, Qwen3 8B, Gemma 4 E4B, and Gemma 4 12B: load time, VRAM usage, output speed, answer quality, and Gemma’s multimodal input. The third article will start from real usage scenarios, not just benchmarks, and ask whether local models can actually improve daily development and material-processing efficiency.

Current Conclusion

For this machine, the most realistic default combination is:

Daily main model: Qwen3 8B
Lightweight tool model: Qwen3 4B
Multimodal experiments: Gemma 4 E4B
High-quality exploration: Gemma 4 12B
Performance baseline: llama.cpp + GGUF
Convenient entry point: Ollama

For long-term use, local model deployment is not as simple as “install a model.” It is more like building a small model workbench on your own machine: where model files live, how the API is exposed, who occupies VRAM, which model is suitable for coding, which one is suitable for summaries, and which one looks good in benchmarks but is unpleasant in daily use. All of this needs to be tested with real tasks.

This article will continue to add measurements later. There is no need to rush to a final conclusion.

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode