VicRes is right but bear in mind that if you change the lock or add a new one, you are now responsible for something that was previously common property.
Also, if the building manager has a valid reason for needing access in a genuine emergency or subject to Tribunal orders, and you have in some way obstructed that access (by not being around when they need to get in, for instance), you could be liable for any damage caused in gaining access.
It all depends on how much you trust the key-holder.
In the building where I live, in the early days, “security” guys were using duplicate pass keys to access apartments they thought were empty – until a woman who would normally have been at work walked out of her shower and caught one of them reading magazines on her lounge.
That said, 15 years on, I can’t count the number of times my spare key – left with the concierge – has rescued me from a night on the street looking for a locksmith