I started looking into it finally after making that comment and came across strata come back here and everyone is saying strata. I guess im trying strata?
need 64gb of system ram. qwen next architecture uses smaller experts than others, and it can keep up with ram swap. I believe it uses a file on disk (ngram) to predict what experts will be needed.
How do you run qwen 3.8 flash next on 16gb vram, if you don’t mind me asking ?
At 3 tokens a second and calling it useble I would guess?
I see people claiming that shit all the time and it’s baffling.
https://sh.itjust.works/comment/28041846
I didn’t believed it until I started using it.
They said 50- i looked up strata, and it uses smaller quantizations/optimizations of that model depending on your system.
I have 48g vram and 192g system ram and get ~15t/s on qwen 3.8 flash next at Q4. Off the top of my head. I think Q6 is 12.
I started looking into it finally after making that comment and came across strata come back here and everyone is saying strata. I guess im trying strata?
https://sh.itjust.works/comment/28041846
Try it out and let me know if you can get higher.
need 64gb of system ram. qwen next architecture uses smaller experts than others, and it can keep up with ram swap. I believe it uses a file on disk (ngram) to predict what experts will be needed.
https://github.com/niko1221/strata
The model I’m using is a Q4. Unsloth UD-IQ4_XS