I have no doubt that they would push for any regulation to advantage them, and they have the resources to achieve that. We still need to try. And the fear of an AI apocalypse isn’t something they just dreamed up recently to make their product look ‘good’ (or powerful, or whatever), nor was it just fished from sci fi. The worries about this scenario were why Altman, Amodei and Musk (among others) got into this business in the first place. They see ASI as inevitable, they sincerely believe it will be powerful enough to destroy humanity, and they each have a messiah-complex which makes them think that only they are intelligent and incorruptible enough to be in charge when it happens.

Of those three premises, I very much doubt any billionaire CEO is incorruptible, but the inevitability of ASI and the power that it would have if it was created are harder to dismiss. If there is even a minuscule chance that they are correct, then AI development should be stopped, until they can prove that it is safe and beneficial to everyone.

Edit: and I take it as a compliment that you only give me a 10% probability of being a bot. :-)

Context matters. In science, if you are proposing something which will upend the established consensus, the burden of proof is on the claimant to show that some detail has been missed, or that some other theory fits the evidence better. What Coxon and others are claiming is not a scientific theory, however. How could anyone provide evidence to prove that building an AGI would likely result in the extinction of the human race? What kind of evidence would be acceptable? And if we continue building bigger and smarter models and before we can gather the evidence it turns out the premise is correct, what then? ’Well, we’re all dead, but at least we stuck to our guns regarding that old axiom.’

I think that, given the potential stakes, the burden of proof should be on the AI companies to show that their technology is not harmful, and until they do they should be restricted from pursuing it. By that I mean that they need to show that their existing models are safe and can be controlled, and if they can’t they need to roll back to the Last Safe Model in their roster. I’d even go further, putting the onus on the companies to show that they are not on net damaging our environment, that their models cannot be compromised to provide terrorists information on building bioweapons, etc., that they will support and help strengthen the vulnerable and never (again) encourage people to commit suicide. And that’s just off the top of my head.

The harms already caused by this technology are staggering, and we need laws to put the brakes on development so that we can collectively adapt, and if the leaders of these companies are offering us a way to do that, let’s grab that opportunity.

That’s why propaganda is so insidious, it provides an alternate reality for people to inhabit so that they don’t need to believe that they’re on the wrong side. For example, if you are presented with two news stories, each true, with one extolling the benefits of what you do while the other points out the drawbacks, it is very hard to take the latter seriously. We automatically go through some very complex mental gymnastics to prove to ourselves that we are good, that what we are doing is good, and that anyone who doesn’t see that is either misguided or corrupt.

People in the big AI labs can be aware of the environmental harm their products are causing, along with the potential social harms, and even that AI might, at some point in the future, kill off humanity, and still convince themself that their decision to work on it is correct, as long as there is any uncertainty about the problems, and any possible benefits from their work. To be human is to be biased, and no amount of intelligence will inoculate you from it.

Why do so many people continue to work with regimes that they fundamentally disagree with? There are lots of reasons, and people are complex so one person can hold several, even contradictory, beliefs at once. Some possible answers;

-It’s easier to go along with it and think of myself as being insignificant to the whole.

-Everyone else is doing it, so it can’t be as bad as I think, right?

-Its too big for any one person to fight/it’s inevitable, so I may as well get something out of it, that way at least some good comes from it.

-If I just focus on my job and what I’m doing right now, I don’t need to worry about the horrible things going on elsewhere or the potential future consequences of what I’m doing.