As the the old saying goes, a verbal agreement isn’t worth the paper it is not written on. A lot of problems arise in Strata Schemes when informal arrangements are made. I’d agree with Fair Trading that the common property remains, and that you shouldn’t touch the fence. Your wish for the common property to return to its original use seems quite reasonable. The options available to you are set out in the dispute resolution procedure available from Fair Trading. The first step is mediation, if that is unsuccessful you can apply for adjudication of the matter. Just do a search for strata dispute resolution and you should find the relevant fact sheet.
There are modest fees for the process, which should not deter you from taking this action. Good luck!