You can’t decline. And the OC has a right of entry to your unit for the purpose of maintaining common property although not for the purpose of stealing your valuables.
If an owner refuses entry, the OC can seek orders granting them entry from NCAT. They can’t just barge in. If the owner still refuses entry in defiance of NCAT orders, then they are in breach of Tribunal orders and can be fined.
The OC or its employees can’t force their way into a unit unless it is a genuine emergency where personal safety or damage to property is immediately at risk.