Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
You must log in or # to comment.
I just bought a 32GB M6 Mac Mini that I’ll also use for local ai. Will probably use Qwen3.8-27B. Accessing it from outside my lan would happen via Tailscale.
I wouldn’t mind running a model at home, mostly to help document my setup. But I just don’t trust it. So no.
Why wouldn’t you trust a local model? I mean sure they are less intelligent but if you dont expose it publicly you should be good to go
I don’t trust it to be accurate; I’d have to reference the sources it pulled from. Extra work for little payoff.
Got it, makes sense. You’d probably have to do some benchmarking first. Depending on how large the source catalog is.
Llama.cpp running qwen 3.8 on a mini PC with 32GB ddr5 ram and 8GB vram. Not the most speedy but it works. I only really access it while at home but I’m also set up with tailscale.
So you’re using CPU offloading? Or the model’s Bonsai2 variation?
A quick glance shows that I’m in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I’m running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I’m happy to have paid a “mere” $2400 compared to what similar systems cost now, and now I’m toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).
Hermes Agent does a lot of random stuff for me right now. It manages my calendar and to-do list in nextcloud via MCP server. It created and maintains an llm-wiki based on my Obsidian Vault. Then it creates a report every morning about both and the weather.
The server has tailscale installed, so yes I can access it anywhere.
I self-host my own neural network, all within the confines of my own skull.
That 33% downtime though
Downtime is the friends made along the way
It’s scheduled maintenance so it doesn’t count toward downtime. Besides, I’m not trying to use it then anyway.
Scheduled most of the time, anyway.
It’s better than github
The downtime isn’t the biggest issue, you can plan around that. The real problem is the random lag on some queries and volatile memory being lost (usually when there’s a context change).
Only 33% downtime? Damn, that must be nice. I really need to get a hardware update.
Me also, and I can barely afford to run that
How much RAM is it using?
Made me think of Johnny Mnemonic. https://www.youtube.com/watch?v=jS2GFqT5ycU
Iirc in a documentary, the brain runs on just 13w of electricity, so the more advanced and efficient AI gets, the less power and therefore resources it’ll need. Maybe Johnny’s world is just that advanced lol
I’m just glad that this is both Top & Hot comment. 🧠✊
If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it’s output into Claude Pro for vetting and it hasn’t found any flaws in the simple scripts and things I’ve asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that’s all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I’m just a little guy.
Even ignoring all of the functional issues with LLMs/“AI” related to their design to never admit to not knowing and preferring to make stuff up instead (hallucinate), I have an ethical qualms about the way they are trained. Two primary issues are that they aggressively scrape data from sources that explicitly do not consent to sharing their work and the data centers which have made hardware impossibly expensive for everyone but them and are so impactful to our underfunded electrical grids that they cause brown outs that keep people from heating their homes among other things, cause major spikes that cause house fires and destroy home appliances, and create an excuse to generate power using the worst polluting generators that we had finally gotten shut down to slow climate change (personally I’m tired of the wildfires and extreme storms from climate change).
So, I personally consider it extremely unethical to use the technology for that and several other reasons which are hard to touch on exhaustively in a post like this.
I’m self-hosting immich (with ML-based facial recognition, semantic search and OCR) and psiTTSa, an offline text-to-speech engine.
(Prob not the answer you were looking for, but a reminder that “AI” is more than just LLMs.)
Does Immich sell/use/keep your ML-based FR? Or is this totally local a never leaves your instance?
That was always the question that bordered me… I run Leeche its a very simple and minimal photo album service and I really like it ! But my family want something that’s more like immich but I’m very cautious about what i’m selfhosting and something that collects data points on my behalf is a no-go !
Immich is entirely free and open-source software and got a grant from the FUTO Foundation, which is fiercely pro privacy. As such, it does not engage in any data sharing. Development is financed through the grant and purchases of the server licence (I got one; it’s ridiculously good software).
I do not run Immich but their privacy policy states:
Locally Stored Data: Immich stores all your photos, albums, settings, and other metadata locally on your device. We do not have access to this data, nor do we transmit or store it on any of our servers.
https://immich.app/privacy-policy
Additionally in their docs I can find no mention of connecting to outside services. I do see references to downloading hugging face models.
https://docs.immich.app/features/facial-recognition#how-face-detection-works
Thank you very much ! I didn’t expected someone doing my part to RTFM o_O !
Without doing again the heavy reading part, what should I expect from hugging face models?
If my reading skills are correct it should be local, but I’m really paranoid about this stuff ^^ Even though I read your posted links, I’m still not sure if I should trust them…
Something written in a manual and what the code realy does are sometimes far appart ^^’
It’s local. You can tell by how your computer dies from inference, when you upload new assets 😅. And you can run the models on another computer separately and watch that one die then, when you upload new assets.
Not only do I run and share it. I’ve built an entire service to allow everyone else to do so safely. https://aihorde.net/
- Qwen3.6 35b a3b Q4_K_M running on llama.cpp
- Open-WebUI as a frontend
- integrations with an embedding model and searxng & crawl4ai
- doing lots of different stuff with it, asking basic stupid questions, recipes, tech stuff, etc
What’s your rig?
1070ti and some hexacore something cpu that’s alright with around 24gb ddr4, I think one of the slower speeds. I also got a faster PC with another llama.cpp instance and I vibe coded a quick and dirty proxy which I can use to switch between the different backends. It’s genuinely useful because I got code editors but also my Open-WebUI using it.
Sounds elaborate but generally it’s all pretty primitive.
Also fyi long preloading steps because my vram is obv not good enough to hold the full model and kv cache and stuff, generation at about 15t/s tops.
My experience is right now we’re at the stage where it’s a good assistant but not great for actually taking over bigger tasks completely.
Am looking forward to:
- experimenting with existing non-uniform quants; I believe those quants are the next step to more bang for your buck on those low-grade consumer rigs
- hoping for new kinds of quants to come out and be more widely adopted
- maybe a more universal or consistent harness that is less prone to broken tool-calls etc.
- better models, specifically a new version for 35b a3b.
Pretty much my setup as well. Though I’m using qwen3.8 27b-Q4_K_M under ollama. Everything running with docker. Add nginx and either a domain+port forwarding or zerotier/tailscale you’ve got an app you can get to from outside your house.
Right now not sure how I’m going to leverage it not probably to streamline my self hosted stuff, help with config mgmt and documentation, web page summarization, etc. we’ll see how it does, just got it setup yesterday
In case you are unaware, friends don’t let friends use ollama. The llama.cpp CLI has become pretty much equally easy to use, plus since you use Docker there’s no need to compile anything.
Edit: typos and missing words
Oh shit I wasn’t aware! Thanks for the heads up, I’ll swap to llama.cpp
Oh damn what a useful link. Gonna check out those orchestrators, maybe I can replace my janky ass proxy.
Very similar, but I’m using Gemma4 12b. I could run 26b but that would exhaust my vram after a short interaction. And ollama. Open WebUI + Home Assistant hooked up with it, a couple very basic scripts I threw together, and connected to Kagi.
I’m trying to get my folks off the Gemini/Google bandwagon for the privacy aspect, but also because it’s just cool from a tech geek perspective. For me, any questions throughout the day that I want info on, but not enough to do a deep-dive for an hour+, gets fed to Steve, my models name. Or I’ll use it as a springboard and then properly read the citations for more info. And I feed it HA automations when I get stuck scratching my head on an edge case or whatever - it’s alright but helps me explain what I want from the automation, and when it understands my goal, great, and when it doesn’t I reflect on what I have setup and can usually go ‘ah ha’ and fix the issue. It’s like a rubber ducky/pair programmer that you can both think is an idiot sometimes, but it doesn’t judge you when you’re the idiot.
That’s a pretty good way to describe those kinds of models. Somewhere between rubber duck debugging and a really lazy intern. Better than an algorithm, faster than a human, but less trustworthy than an intern. It’s great if you know how and when to use it. Maybe it’ll become an intern soon enough, but for now let’s focus on decompiling adobe into bankruptcy.
I’ve got similar running on my RX 7600 XT. It is not a great experience for interactive use, but fine for async tasks. I’m also running nemo-speak for live local dictation and meeting transcription.
Yeah running in the background type beat. Not gonna trust this with anything important but it’s great to have someone just do shit for you or get a second opinion.
Yah doing similar on 5060ti.
That’s gonna be quick af with the same models.
It is, i also use the gemma4b model on it for description generation for my immich photos and its super fast.
Random Question, I’ve been trying to offload some of my stuff handled by claud to basically this exact setup. However one of the things I manage has a pair of MCP servers that are required. I might not be smart enough, but how do you get Qwen to use an MCP?
MCP is a really dumb server that just takes requests and sends back data.
Somewhere in your frontend settings (Open-WebUI has it under “integrations” I believe, some backends can set these things too) you see a section called “MCP server”. You need to keep your MCP server running somewhere (it’s very slim, so not a huge deal usually) and then in those settings you put your IP and port for the server, for example “http://192.168.1.15:8080/”. Usually if you did it right (sometimes you can check if it works in the settings), you can ask the model if it has access to the tools and it will tell you.
What’s happening in the background is that every conversation the AI does, it gets a short preamble with some context about what it can do, and if you connect web search, custom skills or an MCP server (it’s just skills but they are run through their own server basically), the AI sees all that in this preamble if it’s set up correctly. And then when it feels like it needs to use those tools, it will check the description on how to do that, and then it uses them.
Thank you, I’ll give this a go
I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it’s been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.
I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode With MTP, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine higher decode speeds are possible with some tinkering, but I haven’t gotten around to it.
OpenCode recently got hacked and advises changing API keys fyi.
I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit).
What vision model do you use?
How much electricity does it burn? And how much does it cost to let it run overnight? I feel like this is the same problem that the companies have, its way too expensive to use it on a lots of things and it is way cheaper to use it on someone else’s server…
It uses 261 W, probably more expensive than getting a 2nd Mistral subscription with the cost of power where I live (Belgium), but it will be free heat once I have solar panels and a battery.
Thank you, that is not as bad as I expected. 👍
What a timely comment.
I recently gave my kid a gaming PC for his birthday…which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB
Combined with a 5600x3d and 32gb of system memory.
I just started messing with local LLMs yesterday. “Getting started” is rather easy.
pacman -S lm-studio(I use arch, btw), launch it, download a model that your hardware supports, and start talking.Right now I’m working on getting it to talk to vscode.
I used Cline as my VSCode chat plugin for models in LMStudio. Worked pretty well but I have moved my local LLM stack to Hermes in the meantime.
I tried using zoo with qwen3.6 and 2.5 but it failed horrendously. My card could only handle like a 16k context window with those models, and the prompt alone was like 11k.
Tried opencode and that was fast…but can’t get it to interact with system yet. More playing to do.
Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency.
I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using https://pi.dev/
I also started building https://fcli.dev/ to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service.
I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don’t have to worry about token costs, and it often responds faster than Claude would because there usually isn’t a queue on my machines. At this point, I wouldn’t complain if my work provided me 2x B6000 instead of a Claude subscription; I’d still be able to get all my work done.
Would be interested to know your experience with fcli if you’d share more. Just hearing about it.
I mean, I haven’t really started discussing it outside IRL friends, so not surprised you haven’t heard of it 😅
In essence, the problem is twofold:
- LLMs tend to use CLIs more effectively than MCP, at least in the domains I use. However, installing all the CLI tools and configuring them can be cumbersome and not easily repeated. CLIs also cannot be modified, and are often quite verbose.
- LLMs will repeat very similar commands over and over with just a few tweaks. This often happens when there is a gap in a tool or poor documentation.
So fcli is intended to be an easily extended, configured, and authenticated CLI. The “one CLI” to do everything your LLM needs. Ideally, you set up a personal repo to store ‘ucks’ (f packages) in, and you set up credentials for that repo so your LLM can push any new ucks to it. You can then share that repo with other users if they could also benefit from any of that functionality.
As you go along in a session, when you notice the LLM repeatedly running similar verbose commands, you can have it create an uck for it instead and save those tokens.
Likewise, if, say, an MCP server doesn’t have full functionality, you can have your LLM import the SDK from npm into an uck and write a basic fcli wrapper around it to start using.
The other nice thing is that it makes your tool configuration repeatable between LLMs and harnesses. You don’t need to set up your MCP servers in both Claude and Codex and Pi, just in f and tell them to use f.
There’s also a creds uck to make your secrets more portable and (a bit) safer.
In general, I’m trying to model it a bit like pi.dev, but as part of the tooling instead of the harness itself. I think they work quite well together :)
Can you elaborate on LLMs using CLI more effectively than MCP? Do you have any sense of how likely it is just your domain or the MCPs you’re using? Do you have any theories about why that may be?
MCP in general:
- MCP is very verbose (json tool declarations)
- Every tool incurs a prefix cost, which can be in the thousands of tokens across all tools
- Tool outputs are usually dumped straight into the prompt. Sometimes they are truncated and dumped into a file with a warning to the LLM, but usually after they reach thousands of tokens
- Changing any tools breaks your prefix cache.
- Tool discovery can help with 1 a bit, 2 a lot, and 4 a lot, but isn’t a cure-all. It does have its own (much smaller) token overhead too.
CLI in comparison:
- Fairly concise, by nature - short names for frequently used commands, spaces as implicit delimiters, implied outputs.
- Tools are generally implied to be present. A short “this environment contains x,y,z,” system prompt can be very effective. Very minimal prefix cost.
- Tool outputs can be written to a file directly, formatted, head/tailed, grepped, or used in any other way.
- Doesn’t break the prefix cache, but can “confuse” the LLM.
- Tool discovery is implicit in the system. This does have its own overhead, but for many things is implied - ls, find, grep, sed, cat - usually an llm will just assume they’re present, and use them to trim down tool outputs to only the required context. In f, I also added search functionality to avoid parsing ‘help’ messages.
CLI does of course have its own downsides, I think the biggest being security. Give an LLM an MCP server, and all it can do is what that MCP server supports. Give an LLM a Turing complete shell, and it can do anything.
MCP also tends to be harder to set up and harder still to fix when it doesn’t work. I’ve watched Claude fall back to curl commands when a connector doesn’t work many times. I’ve never seen it try curl commands, fail, and then try to set up MCP.
I think this is all fairly domain dependent, but I think my domain (software development and general agenetic assistance) is very common. If you’re using an LLM in a product that you’re selling with a fixed range of features, MCP is probably the way to go. If you’re using an LLM for dynamic functionality with expanding capabilities, I think CLI might fare better.
My main reasoning for all of this is flexibility. In the history of computing, a program has always had a fixed output for a fixed input. LLMs are at the point where they can handle what is effectively an arbitrary input and give an arbitrary output. Giving an LLM a fixed set of tools negates that newfound ability to process the unknown. It’s like giving someone a sledgehammer to bust rocks, but tiring one hand behind their back; the work might get done, but not as efficiently.
I agree with you for common CLIs that the model will natural have in its training set, but when building your own CLI it seems like the self-documenting nature and progressive disclosure of MCP would have a huge advantage.
With a cli you don’t have the prefix cost, but your agent also doesn’t know that the tool exists, what it’s for, how to use it, etc. once you give it that information, then you have the same issues with prefix cost, right? Changing tools does break the prefix cache, but changing tools is a thing you can do. You can’t change (model awareness of) CLI, and if you could then it’d also break the prefix cache, or you’d have tool instructions scattered throughout the context without structure.
Dumping to file is a harness thing, not a MCP thing. A harness will do the same with large terminal outputs.
Likewise setting up a new CLI can be complicated. I never see an agent try to install an mcp, but I also don’t typically see it try to install a new CLI.I don’t see how mcp is any more a fixed set of tools than CLI is. It’s just presented to the agent in a structured way, which is important for discovery and instruction following.
But yeah ease of chaining is certainly useful wrt CLI. That’s a huge win for sure.
I tried, but didn’t want to invest into it, so I used my 12-GB-vram gpu but no matter what I did it was worse than Lumo from proton,which is basically only one LLM from 3rd party that I am okay to use on occasions and since it has really good free tier I gave up on self hosted LLM.
Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track
I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system
I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.
Glm flash (4.something?) does pretty good as well.
I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol
I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.
One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.
Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.
I think it’s cool as fuck. I geek out every time that it works.
Oh, forgot to mention I am running it on a Mac studio with 96 gig of unified memory. I think it’s an M3 ultra chip.
Do what makes you happy mate! Tell the naysayers to fk off.
