Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or # to comment.
I do, but I am becoming increasingly more disappointed as time goes on. Not just self hosted, llms in general. They sometimes help, but they mislead so many times and waste time that you don’t even notice. I think that’s the trap. When you succeed at a task, you become impressed but don’t notice how many times it failed doing a simple task. And as soon as you scratch the surface, you see how you would have done it differently and perhaps in a better way. Even just googling is bad. It does research for you, but it has no critical thinking and can’t decide what is better from the results it gets (other than google ranking) so it often leads you to think it did as good as you would, when it’s nowhere near as good. Every time I did the googling myself after it did, I did it much better. And I mean MUCH better. Ask it to find the app, it misses the most important ones, hallucinates a bunch, for ex. I found this to be the case with frontier models as well.
Self hosting has its benefits, but seeing how the ecosystem looks right now, concluding this is a huge bubble is inevitable. It reminds me of crypto so much. It looks rich and plentiful, but as soon as you dig a mm under the surface - nobody has tested it, it’s got a critical bug, it is overblown and there are issues with no response. No docs, no info, no nothing. For the biggest thing in technology in history, it is awfully hollow. I don’t mean it in a condescending way, in fact community is enthusiastic and very helpful, it’s just that it doesn’t live up to what most would expect.
A caveat I need to mention is I have not used it for coding - I have an irrational fear and resistance towards it, being a programmer. I just won’t touch it, even if it means the end of my career. I’m trying to be grown-up about it, but so far, I dont want to use it, for good and bad reasons.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters Git Popular version control system, primarily for code LTS Long Term Support software version SSH Secure Shell for remote terminal access
3 acronyms in this thread; the most compressed thread commented on today has 9 acronyms.
[Thread #27 for this comm, first seen 25th Jun 2026, 15:40] [FAQ] [Full list] [Contact] [Source code]
Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.
Just coffee?!? That’s cool.
Mine runs on:
- coffee
- spite
- tortilla chips
- & shame
I host my own AI, mostly for testing and because I wanted something that was mine and mine alone. I use Ollama and run models like Llama, Mistral, and Qwen. I honestly don’t use it much, but I wanted to have my own setup just in case online services go down or become less available. It’s part of my whole “own everything I use” mantra that I’ve been on lately.
Running qwen3.6 27b through llama.cpp.
It’s about as capable as sonnet 3.5.
I use it for light scripting, but real coding is done by cloud models.
I’m also using it as the brain for my Hermes agent. It sends me digests of news, subreddits, chats that I’d like to read but don’t have time for. It does a great job researching things on the web for me, too.
That’s a great model and it’s the one I use too.
An aside for anyone reading this:
https://sleepingrobots.com/dreams/stop-using-ollama/
And that barely scratches the surface. Please.
Use anything but Ollama. Even APIs.
Llama.cpp or death!
I’ve tried a few times but with only 8gig of vram it’s simply not worth it.
How much CPU RAM do you have?
64G. But CPU inference is painfully slow.
Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.
You just need to use the right framework, and the right model.
I’d suggest trying ik_llama.cpp and a MoE like one of these: https://huggingface.co/models?other=ik_llama.cpp&sort=modified&search=35B
And speculative decoding like DFlash or MTP (which you can also get specific models for).
EDIT: Wrong link.
I’ll check that out - speed isn’t my biggest issue so much as coding performance… The qwen 3.5 model I was using can write code, but it’s… Meh? Like sometimes it doesn’t even compile.
I did try tweaking llama.cpp to do some cpu offloading and it does seem to allow for much larger contexts at a modest performance loss. I’ll check out larger models.
Try qwen3.6-35b-a3b with a lightweight harness like pi.dev
Having it be able to run commands and try to compile or run the code and see the output helps especially on the “doesn’t compile” part of things
Yeah - I’ve been playing around more with the Qwen3-Coder-30B-A3B-Instruct MoE model and it’s still quite… Meh. I’ve been using llama.cpp and I’ve tried a bunch of tuning. It works and performs well enough (15t/s) but the output is just garbage. I can do some simple coding but I’m finding I’m fighting with it more than if I just wrote the code myself. Maybe I just have standards that are too high. Claude Opus 3.7 is just in an entirely different league…
When you run it, do you use unsloth’s recommended settings for coding?
https://unsloth.ai/docs/models/qwen3.6
Also have preserve thinking on, it helps it stay consistent in multi turn work.
Which model version you’re using can also affect results, usually unsloth’s ones are good.
With all that said, it’s of course a small model so it’s not a super coder. The 27b is better (I’d guess 25-35% better), but of course still a small model so…
So it’ll maybe not be good enough still, but should give it the chance to let it do the best it can :)
