There will be an agreement between you, the landlord and the agent which you both signed.
There may be a clause which permits the agent to act on your behalf in certain circumstances.
If not, then the agent must seek instructions from you and act on what you said.