Information is power. Let the minority of people who take an interest know what’s going on.
Start issuing a newsletter that canvasses the issues that will be coming up at the the AGM and explain why not taking certain action will cost the owners in the value of their properties. In a show of fair comment, invite the bullies to put their point of view in the newsletter.
Add into the minutes of your meetings a record of who voted for and against specific proposals. In other words, expose the bullies not in what they say but in how they vote. And trust your owners to know right from wrong and what’s good and bad for the building.