Timing is important here. If the Manager’s contract is up for renewal and it’s an upcoming AGM then the process can be a little easier. But the disgruntled others need put in the work to find and assess another, and then to put them up for a comparative vote as you will likely need more than one. You can explain the merits of your preferred Manager in the lead up supporting stuff to your Motion that is sent out and it essentially then becomes just another vote on the day. This gives time for folks to get their heads around it, vote early, and even avoid any in-depth discussion on the day.
But if the contract still has time to go rather than first trying to remove them you might try a vote of no confidence (can you do that?), or just a discussion directly or ideally via the committee, that they haven’t appeared up to the task, and hope that is enough inspiration for them to try a little harder.