If you have permission to keep your cats then you should have no concerns that they will be taken away from you.
If the owner who lives opposite you purchased after you obtained permission to keep the cats then he should have done his due diligence and searched the records before he purchased his property. He would have seen that your Lot contained the cats. At that time he could have chosen not to buy the property that he ended up purchasing. He didn’t.
Alternatively, if the owner who lives opposite you purchased his property before you obtained permission for your cats and is only now interested in your cats and his allergies then he had an option to vote against permission to keep the cats three years ago. If he did then he was in the minority.