If you ultimately want to have the fence demolished, then not saying anything about it until some later date is not going to help your case. As Whale has pointed out, doing things on a wink and a nod can have very bad consequences down the line.
Going back to your original post, the DOFT is correct, the land is not theirs but in giving them permission to build the fence you have effectively given them control of that land, and a bigger backyard, for nothing. A subsequent purchaser of your lot (or theirs for that matter) might not be very happy about that.
The reason why a subsequent purchaser of their lot might not be happy is because while the fence is there they have appropriated common property with nothing legal to back it up, and if I were looking at buying the lot that would be an issue.
I appreciate you don’t want things to descend into warfare, but you need to sort it out, both the parking issue and the bigger issue of the fence.