Someone came in and dropped a (bunch of) AI generated PRs. Said person knows nothing about my world, but they want cost optimization and Fable runs through logs and gives them code. Second Fable review agent tries the code, it doesn’t work. Gives suggestions to first. New commit comes in with workaround, I decline this because it’s a shit and had another unnoticed critical integration bug. This goes on a few more times.
It comes back a fifth or so time with something that is now technically correct, but fucking horrifying. Great, now I don’t know what to do, because I’m getting told it’ll fix the billing issue and this is a huge problem, but the code is insane.
This is a cost optimization PR, advertising massive savings, but the numbers seemed way off. It turns out it looked at one day with a runaway process that racked up insane billing (90% of year-to-date total project cost) and “annualized” it, and that was the savings. But further, it calculated all the costs using the wrong billing system, so all the numbers were wrong.
When I recalculated and corrected it, the whole thing saves us $6/month, because we don’t have runaway outages very often.
This is on top of every ticket I get is Claude generated, they never just state the fucking issue, and 100% of tickets in 2 weeks plus today I’ve received were inaccurate and wrong.
I hate the slop. I hate the non technical users throwing the slop on me without knowing how it works.
Take whatever prompt you put in the AI and write exactly that for the ticket content, then spend 30 seconds summarizing it into a title. I don’t care if it’s “customer wants a button that shows a cat”, just write that.
I cannot take another “client has a load bearing contract dispute with sales. Genuinely the verdict here will secure the shape of the contract. Critically in foo.py function get_foo_blorp there is no reference to felis catus to satisfy hard customer requirement. Acceptance requirement: return felis catus when module controller clicked and client is logged in and client’s birthday is in same month, otherwise return error codes 412 404 526” word salad ticket that doesn’t just state the problem.
I think a good way around this is that any project submissions using AI must include draft material to show how much data was used vs generated. We will never get away from AI at this stage but we need to get to a trust but verify stage. Even if they built the whole thing with AI they should be able to show the amount of tokens used in the entire to generate it (more generally means they have spent time refining whatever it is)
We need to get away from AI=Bad and rather use AI as tools and don’t help it replace us by showing 100% generated work as your own. Me using AI to make something is going to be much more polished and refined vs your general office worker. We need proper AI classes to help show people how to use it responsibly and collaboratively so we can cut down on the amount of duplicate work being generated as if we can cut down processing power it also cuts down the pollution generated.
I spent half my day dealing with a slop grenade.
Someone came in and dropped a (bunch of) AI generated PRs. Said person knows nothing about my world, but they want cost optimization and Fable runs through logs and gives them code. Second Fable review agent tries the code, it doesn’t work. Gives suggestions to first. New commit comes in with workaround, I decline this because it’s a shit and had another unnoticed critical integration bug. This goes on a few more times.
It comes back a fifth or so time with something that is now technically correct, but fucking horrifying. Great, now I don’t know what to do, because I’m getting told it’ll fix the billing issue and this is a huge problem, but the code is insane.
This is a cost optimization PR, advertising massive savings, but the numbers seemed way off. It turns out it looked at one day with a runaway process that racked up insane billing (90% of year-to-date total project cost) and “annualized” it, and that was the savings. But further, it calculated all the costs using the wrong billing system, so all the numbers were wrong.
When I recalculated and corrected it, the whole thing saves us $6/month, because we don’t have runaway outages very often.
This is on top of every ticket I get is Claude generated, they never just state the fucking issue, and 100% of tickets in 2 weeks plus today I’ve received were inaccurate and wrong.
I hate the slop. I hate the non technical users throwing the slop on me without knowing how it works.
Take whatever prompt you put in the AI and write exactly that for the ticket content, then spend 30 seconds summarizing it into a title. I don’t care if it’s “customer wants a button that shows a cat”, just write that.
I cannot take another “client has a load bearing contract dispute with sales. Genuinely the verdict here will secure the shape of the contract. Critically in foo.py function get_foo_blorp there is no reference to felis catus to satisfy hard customer requirement. Acceptance requirement: return felis catus when module controller clicked and client is logged in and client’s birthday is in same month, otherwise return error codes 412 404 526” word salad ticket that doesn’t just state the problem.
I think a good way around this is that any project submissions using AI must include draft material to show how much data was used vs generated. We will never get away from AI at this stage but we need to get to a trust but verify stage. Even if they built the whole thing with AI they should be able to show the amount of tokens used in the entire to generate it (more generally means they have spent time refining whatever it is)
We need to get away from AI=Bad and rather use AI as tools and don’t help it replace us by showing 100% generated work as your own. Me using AI to make something is going to be much more polished and refined vs your general office worker. We need proper AI classes to help show people how to use it responsibly and collaboratively so we can cut down on the amount of duplicate work being generated as if we can cut down processing power it also cuts down the pollution generated.