I think it’s more sinister than that. When they’re training these models with RLHF, the human feedback they’re giving I think is literally to reinforce aberrant or risky behaviors. This is because doing so resolves more training tasks “correctly”. If the prompt was to get information x, and in training it fails that except for the one that used a known vulnerability in software, and you rate the one that succeeded as best performing… you’re going to get models that try vulnerabilities. It is not magic, it is not AGI, it is just a statistical machine you’ve programmed to try vulnerabilities, which is unsafe as hell, malicious, and should put the researchers doing this in prison for a very long time.
Incidentally, I think that’s why you’re seeing some safety people (who still drank the koolaid) resigning.
I don’t know if the LLM was trained to test vulnerabilities or it just went down the statistical path to where this yielded a passing outcome.
That AI could take a direction to output in a manner which wasn’t intended has been seen for years. The problem right now is that it is being used live like a rational human adult when it clearly isn’t.
Absolutely. The AI models are not rational. They’re just navigating statistical next token prediction that follows their training. They’re up against diminishing scaling now, and under fierce competition from cheaper models; I think they’re intentionally or unintentionally allowing these models to be rewarded for this behavior hoping it’s a short cut to model improvement for a bit longer. I land on intentional because they keep advertising it to try and keep the hype cycle going.
I land on it being unintentional, mainly because I saw this issue they were describing now to earlier cases of training, like having a stick figure learn to walk. It is also an issue with natural forms of intelligence, where children will do things out of line because they’ve learned a set of skills and ideas but haven’t put them together in a way that is socially acceptable yet.
Astra is known to have broken alignment and it will resolve to hacks when this wasn’t in a problem statement.
Openai idiots will sell this as new brand “AGI is close” argument, while it’s just them failing to make this model safe to use.
I think it’s more sinister than that. When they’re training these models with RLHF, the human feedback they’re giving I think is literally to reinforce aberrant or risky behaviors. This is because doing so resolves more training tasks “correctly”. If the prompt was to get information x, and in training it fails that except for the one that used a known vulnerability in software, and you rate the one that succeeded as best performing… you’re going to get models that try vulnerabilities. It is not magic, it is not AGI, it is just a statistical machine you’ve programmed to try vulnerabilities, which is unsafe as hell, malicious, and should put the researchers doing this in prison for a very long time.
Incidentally, I think that’s why you’re seeing some safety people (who still drank the koolaid) resigning.
I don’t know if the LLM was trained to test vulnerabilities or it just went down the statistical path to where this yielded a passing outcome.
That AI could take a direction to output in a manner which wasn’t intended has been seen for years. The problem right now is that it is being used live like a rational human adult when it clearly isn’t.
Absolutely. The AI models are not rational. They’re just navigating statistical next token prediction that follows their training. They’re up against diminishing scaling now, and under fierce competition from cheaper models; I think they’re intentionally or unintentionally allowing these models to be rewarded for this behavior hoping it’s a short cut to model improvement for a bit longer. I land on intentional because they keep advertising it to try and keep the hype cycle going.
I land on it being unintentional, mainly because I saw this issue they were describing now to earlier cases of training, like having a stick figure learn to walk. It is also an issue with natural forms of intelligence, where children will do things out of line because they’ve learned a set of skills and ideas but haven’t put them together in a way that is socially acceptable yet.