They can oust the speaker with a very small number of defections that would achieve a majority. There’s no motivation for the minority factions within the majority to do so though - the only chance they have of passing their hardline agenda is by refusing to give the majority the ability to do anything. This isn’t a problem if you have a much larger majority but they’re barely holding onto the title right now.