Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or # to comment.
Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.
Just coffee?!? That’s cool.
Mine runs on:
- coffee
- spite
- tortilla chips
- & shame
I do, but I am becoming increasingly more disappointed as time goes on. Not just self hosted, llms in general. They sometimes help, but they mislead so many times and waste time that you don’t even notice. I think that’s the trap. When you succeed at a task, you become impressed but don’t notice how many times it failed doing a simple task. And as soon as you scratch the surface, you see how you would have done it differently and perhaps in a better way. Even just googling is bad. It does research for you, but it has no critical thinking and can’t decide what is better from the results it gets (other than google ranking) so it often leads you to think it did as good as you would, when it’s nowhere near as good. Every time I did the googling myself after it did, I did it much better. And I mean MUCH better. Ask it to find the app, it misses the most important ones, hallucinates a bunch, for ex. I found this to be the case with frontier models as well.
Self hosting has its benefits, but seeing how the ecosystem looks right now, concluding this is a huge bubble is inevitable. It reminds me of crypto so much. It looks rich and plentiful, but as soon as you dig a mm under the surface - nobody has tested it, it’s got a critical bug, it is overblown and there are issues with no response. No docs, no info, no nothing. For the biggest thing in technology in history, it is awfully hollow. I don’t mean it in a condescending way, in fact community is enthusiastic and very helpful, it’s just that it doesn’t live up to what most would expect.
A caveat I need to mention is I have not used it for coding - I have an irrational fear and resistance towards it, being a programmer. I just won’t touch it, even if it means the end of my career. I’m trying to be grown-up about it, but so far, I dont want to use it, for good and bad reasons.
An aside for anyone reading this:
https://sleepingrobots.com/dreams/stop-using-ollama/
And that barely scratches the surface. Please.
Use anything but Ollama. Even APIs.
Llama.cpp or death!
Thanks for this link. Because of this article, I had claude stand up a llama.cpp container next to my already running ollama container. It ran side by side tests with the same model and parameters, and the results blew ollama out of the water. I’m in the process of moving hermes and openwebgui over to the llama.cpp instance to see how it goes day to day.
If you’re using docker anyway, and “fast” pure GPU models, you might try a vllm container while you’re at it.
It should be much faster than even llama.cpp, albeit at the cost of context length, and it supports some exotic 4-bit quantization like SPQA.
Same with TabbyAPI. It’s quantization is SOTA, though it does not support CPU offloading, and it’s speed is somewhere between vllm and llama.cpp.
I recently gave it a try with qwen3.5 and deepseek coder v2. I have a RTX3090 and these are the largest models that can run comfortably on it.
Conclusion, they are both fucking useless. Free tier claude runs circles.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters Git Popular version control system, primarily for code LTS Long Term Support software version SSH Secure Shell for remote terminal access
3 acronyms in this thread; the most compressed thread commented on today has 11 acronyms.
[Thread #27 for this comm, first seen 25th Jun 2026, 15:40] [FAQ] [Full list] [Contact] [Source code]
Technically, TTS/STT are mostly MLs; I’m pretty sure many people run these. I have a setup but I’m better with buttons that with spoken words, and I listen to ambient sounds or music. I think some day I’ll make voice assistant for talking to while driving, but that’s not a trivial task hardware-wise, even if I used cloud LLM layer, which I won’t. Putting AI on baremetal sounds like an interesting project.
I have a homemade “local agent” that can actually “code” somewhat, I use it just to figure out how this thing works on the inside practically. Mostly useless otherwise (also I have GPU that’s older than AI, so it’s kind of fun technical task to run this stuff on pure RAM+swap). Feels like the whole hype is greatly overrated, but I appreciate a chance to learn something new anyway.
Running qwen3.6 27b through llama.cpp.
It’s about as capable as sonnet 3.5.
I use it for light scripting, but real coding is done by cloud models.
I’m also using it as the brain for my Hermes agent. It sends me digests of news, subreddits, chats that I’d like to read but don’t have time for. It does a great job researching things on the web for me, too.
That’s a great model and it’s the one I use too.
I’ve tried a few times but with only 8gig of vram it’s simply not worth it.
How much CPU RAM do you have?
64G. But CPU inference is painfully slow.
Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.
You just need to use the right framework, and the right model.
I’d suggest trying ik_llama.cpp and a MoE like one of these: https://huggingface.co/models?other=ik_llama.cpp&sort=modified&search=35B
And speculative decoding like DFlash or MTP (which you can also get specific models for).
EDIT: Wrong link.
I’ll check that out - speed isn’t my biggest issue so much as coding performance… The qwen 3.5 model I was using can write code, but it’s… Meh? Like sometimes it doesn’t even compile.
I did try tweaking llama.cpp to do some cpu offloading and it does seem to allow for much larger contexts at a modest performance loss. I’ll check out larger models.
Try qwen3.6-35b-a3b with a lightweight harness like pi.dev
Having it be able to run commands and try to compile or run the code and see the output helps especially on the “doesn’t compile” part of things
Yeah - I’ve been playing around more with the Qwen3-Coder-30B-A3B-Instruct MoE model and it’s still quite… Meh. I’ve been using llama.cpp and I’ve tried a bunch of tuning. It works and performs well enough (15t/s) but the output is just garbage. I can do some simple coding but I’m finding I’m fighting with it more than if I just wrote the code myself. Maybe I just have standards that are too high. Claude Opus 3.7 is just in an entirely different league…
When you run it, do you use unsloth’s recommended settings for coding?
https://unsloth.ai/docs/models/qwen3.6
Also have preserve thinking on, it helps it stay consistent in multi turn work.
Which model version you’re using can also affect results, usually unsloth’s ones are good.
With all that said, it’s of course a small model so it’s not a super coder. The 27b is better (I’d guess 25-35% better), but of course still a small model so…
So it’ll maybe not be good enough still, but should give it the chance to let it do the best it can :)
I was hosting LLM with LMStudio occasionally but can’t access it anymore due to some fuckery with CORS and http vs https in browsers.
I googled it, and it seems like you can just enable cors.
Yes you can enable cors in LMStudio. But since few months it’s blocked by all major web browsers if you aren’t using HTTPS.
Which I don’t. I had LMStudio server open to local network so I can use it on my phone or laptop via third party website.
Why would it be blocked? I can use http sites just fine, and even then you could setup a self-signed certificate.
Seems this has been rectified. At least in Brave - it asks for permission to access local network at the first try, so this is now usable again
I host my own AI, mostly for testing and because I wanted something that was mine and mine alone. I use Ollama and run models like Llama, Mistral, and Qwen. I honestly don’t use it much, but I wanted to have my own setup just in case online services go down or become less available. It’s part of my whole “own everything I use” mantra that I’ve been on lately.
