Local LLM Takes 3 Minutes to Say Hello? Fixing Slow Ollama on an AMD RX 6800 XT (Vulkan to ROCm)

A local model taking two to three minutes to answer "introduce yourself in one sentence" is not normal, even on a mid-range GPU. On my AMD Radeon RX 6800 XT the cause turned out to be two stacked problems: the agent app was sending a ~20,000-token prompt with every message, and Ollama had silently fallen back from ROCm to Vulkan, where long prompts are processed very slowly. Fixing the second one took a 25,000-token prompt from about 9 minutes to about 30 seconds.

Everything below was measured on one Windows 11 machine on 2026-09-28 with Ollama 0.34.4 and qwen3:8b / qwen3:14b (Q4_K_M). Your numbers will differ; the diagnostic steps will not.

The symptom

• A desktop AI agent pointed at local Ollama showed "Worked for 2m 53s" for a one-line greeting.
• Cloud models in the same app answered in seconds, so the app itself looked fine.
• Short prompts typed directly into ollama run felt fast, which made the slowness look random.

Before: a one-line greeting took 2 m 8 s and 2 m 53 s on the local model.

Step 1: split the time into prompt processing and generation

Ollama's native API returns timing fields for every request. Send one request and divide:

curl -s localhost:11434/api/chat -d '{"model":"qwen3:8b","stream":false,
  "messages":[{"role":"user","content":"hi"}]}'
# prompt tokens/s = prompt_eval_count / (prompt_eval_duration / 1e9)
# generation tokens/s = eval_count / (eval_duration / 1e9)

The server log (%LOCALAPPDATA%\Ollama\server.log on Windows) prints the same numbers per request, plus the prompt size in task.n_tokens. That log told the real story:

Measurement	Value	Verdict
Generation speed	50–67 tokens/s	Normal for an 8B model on this card
Prompt size sent by the agent app	14,000–19,500 tokens per message	System prompt, tool definitions and history
Prompt processing speed	~21 tokens/s	Far too slow — this is the bottleneck

Generation was fine. Reading the prompt was not. Cloud APIs chew through 20k tokens in a second or two, so the same app never looked slow with them.

One more trap: with the default OLLAMA_NUM_PARALLEL=1, requests queue. A tiny test request that computed in 0.2 seconds still took 60 seconds to return because it waited behind the app's 20k-token request. If you benchmark while an app is running, you are measuring the queue.

Step 2: check which GPU backend Ollama actually uses

Search the server log for inference compute:

msg="inference compute" library=Vulkan ... description="AMD Radeon RX 6800 XT"

library=Vulkan means Ollama is using the portable Vulkan backend, not AMD's ROCm. The RX 6800 XT (gfx1030) is listed as a ROCm-supported card in Ollama's GPU documentation, so this is a fallback, not a design choice.

Before changing backends I tried the cheaper fix. My environment had OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 set (both documented in the Ollama FAQ). Removing them on Vulkan roughly doubled prompt speed, but it still collapsed as the prompt grew:

Prompt length (Vulkan, FA off)	Prompt processing
1,195 tokens	563 tokens/s
5,070 tokens	212 tokens/s
25,754 tokens	48 tokens/s (≈ 9 minutes total)

Step 3: find out why ROCm is not detected

Start Ollama with Vulkan disabled and debug logging on, and read the GPU discovery lines:

$env:OLLAMA_VULKAN='0'; $env:OLLAMA_DEBUG='1'; $env:AMD_LOG_LEVEL='3'
ollama serve

Two lines stood out:

WARN msg="AMD driver is too old. Update your AMD driver to enable GPU inference."
hipGetDeviceCount: Returned hipErrorNoDevice

Do not take the first line at face value. The installed driver was Adrenalin 26.8.1, dated 2026-08-17 — current. The registry key under HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e968-...} shows RadeonSoftwareVersion if you want to confirm yours.

The useful test is to ask each HIP runtime directly how many GPUs it sees. A few lines of Python with ctypes are enough:

import ctypes, os
dll = r"C:\path\to\amdhip64_7.dll"
os.add_dll_directory(os.path.dirname(dll))
h = ctypes.CDLL(dll)
n = ctypes.c_int(-1)
print(h.hipGetDeviceCount(ctypes.byref(n)), n.value)   # 0, 1 means "found one GPU"

HIP runtime	Source	GPUs found
HIP 6 (amdhip64_6.dll)	Installed by the AMD driver in System32	1
HIP 7.1 (amdhip64_7.dll)	Bundled with Ollama 0.34.4	0
HIP 7.14 (amdhip64_7.dll)	Bundled with llama.cpp release b10766	1

So the driver and the card were fine. Only the HIP 7.1 runtime that ships inside this Ollama build failed to enumerate the GPU.

Step 4: confirm ROCm is actually faster before touching Ollama

llama.cpp publishes Windows ROCm builds on its releases page. llama-bench can load the exact GGUF file Ollama already downloaded (it lives under %USERPROFILE%\.ollama\models\blobs), so the comparison is apples to apples. Prompt processing of 1,024 tokens on top of 16,384 tokens of existing context:

Backend	Flash attention	Prompt @ 16k context	Generation @ 16k
Vulkan	off	79 tokens/s	27.5 tokens/s
Vulkan	on	63 tokens/s	49.3 tokens/s
ROCm	off	127 tokens/s	17.2 tokens/s
ROCm	on	687 tokens/s	50.0 tokens/s

ROCm with flash attention was roughly 9–11× faster on long prompts. Note that flash attention hurt on Vulkan and helped enormously on ROCm — the right setting depends on the backend.

Step 5: give Ollama a HIP runtime that sees the GPU

This is a workaround, not an official fix. Back up first.

1. Quit Ollama completely (tray icon → Quit).
2. Open %LOCALAPPDATA%\Programs\Ollama\lib\ollama\rocm_v7_1\ and copy amdhip64_7.dll and the amd_comgr*.dll files into a backup folder.
3. From the llama.cpp win-rocm-7.14 zip, copy amdhip64_7.dll, amd_comgr.dll and rocm_kpack.dll into rocm_v7_1\.
4. Set OLLAMA_FLASH_ATTENTION=1 as a user environment variable and start Ollama again.

The "driver is too old" warning still prints, but discovery now reports:

msg="inference compute" library=ROCm compute=gfx1030 name=ROCm0 description="AMD Radeon RX 6800 XT"

An Ollama update will overwrite these files. If things get slow again after an update, check that log line first.

Verification: the same prompt through Ollama

Setup	25.7k-token prompt	Total time	One-line greeting
qwen3:8b, Vulkan	48 tokens/s	≈ 544 s	—
qwen3:8b, ROCm + FA	796 tokens/s	41.5 s (incl. 8 s model load)	1.6 s
qwen3:14b, ROCm + FA, f16 cache	301 tokens/s	109 s	3.4 s
qwen3:14b, ROCm + FA, q8_0 cache	475 tokens/s	69 s	—

After: 14 s and 27 s for the same greeting. The 27 s reply was the first message after switching to qwen3:14b, so the whole ~12k-token prompt had to be processed from scratch.

Bonus: why the bigger model felt unusable before

With a 40,960-token context and the default f16 KV cache, qwen3:14b used 15.7 GB of the card's 16 GB. Generation after a long prompt dropped to 4 tokens/s. Setting OLLAMA_KV_CACHE_TYPE=q8_0 halved the cache, brought usage down to 12.6 GB and generation back to 26.5 tokens/s. If a larger model makes the whole PC struggle, check ollama ps: when the size is at the edge of your VRAM, shrink the cache or the context before blaming the model.

In the agent app itself, the first message after switching models took 27 seconds, because the cache built by the previous model cannot be reused and all ~12k prompt tokens had to be read again. The next message in the same conversation only needed to process the 1,650 new tokens (3.5 s).

Checklist

• Split latency into prompt processing and generation before changing anything.
• Check how many tokens your app really sends per message — agent apps often send 10–20k.
• Look for library= in the inference compute log line to see which backend is in use.
• Do not trust "driver is too old" on its own; check the driver version and test each HIP runtime directly.
• Benchmark flash attention per backend — it helped on ROCm and hurt on Vulkan here.
• Keep model + KV cache comfortably under your VRAM; q8_0 cache is the easiest win for larger models.