I just recently installed Strata which runs on qwen 3.8 flash next. I’m getting over 50 toks. Now a deepseek v4 version is out along with glm 5.3 flash. It’s slower but useable and very capable. All this on a 5080 16GB and 128GB ddr5.
The future for me is going to be open weights and on your own hardware.
I mean that sounds great and I’m happy for you, but you’re working with hardware that costs more than most people on here’s cars when it’s available at all. How viable would your workflow be on an 8gb 1080ti with 32gb of ddr4?
Running locally makes little sense anyway because computing multiple replies at once costs the same as computing one. A home setup isn’t going to get utilization anywhere close to optimal.
At no point did I say it was man. Just not something most people have access to. “Got to have money to make it” painfully true as usual. I have $234 in my wallet, a late 90s Honda to crash in, and sheer force of will to try and change that (it’s not working lol). The 32GB/1080ti example is the nicest computer any of my friends own, my own rig is a scavenged parts pile centered around a 10th gen i7 that will be complete when some ram falls into my lap and right now I’m posting from a phone that was $150 6 years ago.
You know, that’s fair. Most people do not need a rig like mine. I would still be using my 8 year old computer if my new clients data wasn’t so sensitive that I have to run local inference.
As someone who was also broke and lived in an old Honda, I know what you’re going through and I’ve been there.
If you truly want to change your outcome, you gotta find a job that has high growth potential and a low barrier for entry. Personally I worked at a bank for almost a decade. Started as a teller(low barrier,no college needed), ended as a regional manager(high growth). I hated every minute of it. But it gave me a chance to save up and find a career I enjoy.
Good luck my dude. You will be successful. I believe in you.
That’s how it needs to be, and how it should be. Otherwise, we are all permanently screwed by big tech, subscribed and licensed to even own a computer by 2035; MicroSlop says open-source AI is good somehow, but we will see through their actions if they lock down their new line of NVIDIA Spark laptops to an approved number of models. I heard rumours that Deekseek might be banned in America; I can’t see how that would work though.
Deepseek is open weights. You can just download and use it locally. I would never run deepseek using their servers. Or z.ai servers. That I agree with China is still pretty bad.
Nice setup. I’m running Qwen3.8 27B at 17GB with a 32k context window on a 3090. It does most of what I need for everyday use and its all local, private, no internet connection needed.
The gap between running an LLM locally and cloud API is closing fast.
need 64gb of system ram. qwen next architecture uses smaller experts than others, and it can keep up with ram swap. I believe it uses a file on disk (ngram) to predict what experts will be needed.
I started looking into it finally after making that comment and came across strata come back here and everyone is saying strata. I guess im trying strata?
Running on your hardware is always going to be less efficient than running in a data center. You can run small models on green data centers - best of both worlds (unless you need privacy then run local).
Ever since I switched to Cortecs I’m spending like 20 cents / month for my LLM usage cuz I mostly use small models and try using them responsibly.
I just recently installed Strata which runs on qwen 3.8 flash next. I’m getting over 50 toks. Now a deepseek v4 version is out along with glm 5.3 flash. It’s slower but useable and very capable. All this on a 5080 16GB and 128GB ddr5.
The future for me is going to be open weights and on your own hardware.
I mean that sounds great and I’m happy for you, but you’re working with hardware that costs more than most people on here’s cars when it’s available at all. How viable would your workflow be on an 8gb 1080ti with 32gb of ddr4?
Running locally makes little sense anyway because computing multiple replies at once costs the same as computing one. A home setup isn’t going to get utilization anywhere close to optimal.
I mean, it’s not really a toy. It’s a business expense. I use it everyday to make money.
At no point did I say it was man. Just not something most people have access to. “Got to have money to make it” painfully true as usual. I have $234 in my wallet, a late 90s Honda to crash in, and sheer force of will to try and change that (it’s not working lol). The 32GB/1080ti example is the nicest computer any of my friends own, my own rig is a scavenged parts pile centered around a 10th gen i7 that will be complete when some ram falls into my lap and right now I’m posting from a phone that was $150 6 years ago.
You know, that’s fair. Most people do not need a rig like mine. I would still be using my 8 year old computer if my new clients data wasn’t so sensitive that I have to run local inference.
As someone who was also broke and lived in an old Honda, I know what you’re going through and I’ve been there. If you truly want to change your outcome, you gotta find a job that has high growth potential and a low barrier for entry. Personally I worked at a bank for almost a decade. Started as a teller(low barrier,no college needed), ended as a regional manager(high growth). I hated every minute of it. But it gave me a chance to save up and find a career I enjoy.
Good luck my dude. You will be successful. I believe in you.
That’s how it needs to be, and how it should be. Otherwise, we are all permanently screwed by big tech, subscribed and licensed to even own a computer by 2035; MicroSlop says open-source AI is good somehow, but we will see through their actions if they lock down their new line of NVIDIA Spark laptops to an approved number of models. I heard rumours that Deekseek might be banned in America; I can’t see how that would work though.
Deepseek is open weights. You can just download and use it locally. I would never run deepseek using their servers. Or z.ai servers. That I agree with China is still pretty bad.
Nice setup. I’m running Qwen3.8 27B at 17GB with a 32k context window on a 3090. It does most of what I need for everyday use and its all local, private, no internet connection needed.
The gap between running an LLM locally and cloud API is closing fast.
Yeah but getting your own hardware is getting expensive too.
Only because there is artificial scarcity in play
There’s plenty of actual scarcity to go around too
How do you run qwen 3.8 flash next on 16gb vram, if you don’t mind me asking ?
need 64gb of system ram. qwen next architecture uses smaller experts than others, and it can keep up with ram swap. I believe it uses a file on disk (ngram) to predict what experts will be needed.
At 3 tokens a second and calling it useble I would guess?
I see people claiming that shit all the time and it’s baffling.
https://sh.itjust.works/comment/28041846
I didn’t believed it until I started using it.
They said 50- i looked up strata, and it uses smaller quantizations/optimizations of that model depending on your system.
I have 48g vram and 192g system ram and get ~15t/s on qwen 3.8 flash next at Q4. Off the top of my head. I think Q6 is 12.
I started looking into it finally after making that comment and came across strata come back here and everyone is saying strata. I guess im trying strata?
https://sh.itjust.works/comment/28041846
Try it out and let me know if you can get higher.
https://github.com/niko1221/strata
The model I’m using is a Q4. Unsloth UD-IQ4_XS
Running on your hardware is always going to be less efficient than running in a data center. You can run small models on green data centers - best of both worlds (unless you need privacy then run local).
Ever since I switched to Cortecs I’m spending like 20 cents / month for my LLM usage cuz I mostly use small models and try using them responsibly.
That’s why you do it at work on a secured laptop you own that’s on their network. Modern problems. They pay electricity and all the jazzzzzzzz.