We’ve fitted Qwen3.8-27B and a 240K-token context allocation onto a single RTX 3090 using llama.cpp. That’s 245,760 tokens, with quantized weights and conversation cache—not a full 256K.
The interesting part is the combination: substantial context capacity, useful generation speed, and CUDA on an older consumer GPU. The catch is how long a fresh, large prompt takes to read.
Why this stack?
Qwen3.8-27B is a practical model size for one GPU. We’re using UD-Q4_K_XL quantization to leave room for context in memory. It passed our targeted JSON, tool-call-format, and long-context retrieval checks. That makes it worth testing for assistant work, not a proven replacement for a frontier model.
The RTX 3090 combines 24 GB of VRAM with CUDA. Even the newer RTX 5080 has only 16 GB. A well-priced used 3090 can therefore be attractive for local inference—not merely a way to recycle hardware. Intel’s Arc Pro cards offer competing memory capacity, but without CUDA. We haven’t benchmarked those alternatives; our argument is capacity, measured speed, and software support together. Used price, condition, and cooling still matter.
llama.cpp provides CUDA acceleration, an OpenAI-compatible server, and direct control over quantization, memory, and speculative decoding. Those controls matter when the model and its context cache have to share one card.
What we’re getting
Median generation speeds in our repeated tests:
- 4K prompt: about 58 tokens/sec fresh; 81 on a cached follow-up.
- 64K prompt: about 44 tokens/sec fresh; 62 on a cached follow-up.
- 128K prompt: about 35 tokens/sec fresh.
These tests used 512-token outputs and one active request. Cached follow-ups reused almost the entire prompt prefix.
The catch: the fresh 128K request took about 191 seconds to produce its first output token. Once generation starts, 35 tokens/sec is usable. Waiting for the model to read the input is another experience entirely.
The limits
Our 240K allocation uses a Q4_0 KV cache and one concurrent slot. We tested throughput only through 128K: fitting a context allocation in memory doesn’t establish speed or reasoning quality across its full capacity.
There isn’t much tuning headroom, either. Increasing the processing microbatch from 256 to 512 caused a CUDA out-of-memory failure. Reducing speculative draft depth from four tokens to three helped some workloads but hurt cached 64K performance. Bigger settings aren’t automatically faster.
My takeaway: a well-priced 3090 remains an interesting local-inference card because it combines memory capacity with CUDA—not because it’s new. This stack is useful for ongoing conversations and bounded tasks. Reuse context, trim irrelevant input, and measure time to first token alongside generation speed.
It can hold a lot more text than you’d necessarily enjoy asking it to read from scratch.
Configuration: llama.cpp 050dde50c9d7, CUDA, flash attention, UD-Q4_K_XL weights, Q4_0 K/V cache, MTP draft depth 3, microbatch 256. Results describe this configuration, not a general model-quality ranking.