How Fast Can DeepSeek Run on 8GB VRAM?
So we left with an answer to a question but many more questions to answer, which is good for curious people.
If 3.5 tok/sec is not the wall, what is?
Are you telling me that on 8GB of VRAM and 24GB of RAM, with disk streaming, you can use DeepSeek? What is the maximum speed? Where is the bottleneck? What can be improved and why does it improve?
The answer is very nuanced.
Let's begin by the last questions.
Can stuff be improved?
Again, you have gigahertz of computation on basically every modern PC. They are very capable machines. What is often missing is optimization and developing useful strategies and tricks based on the platform and constrained resources you actually have. Do you know your:
- PCIe transfer speed?
- RAM speed?
- transfer speed from disk to RAM?
- transfer time from RAM to GPU?
- transfer time from disk to GPU?
When you...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
