If you own the apartment of course you can. Just get the barrel rekeyed and only give a spare key to a trusted person who will respect your wishes regarding access.
If you’re a tenant then you’ll need to get permission from your land lord through the agent (if applicable).
However, in the first instance retrieve the keys from the building manager with an assurance that they haven’t been copied. Once returned let them and their superior (maybe even the whole BC) know that the BM didn’t follow your instructions regarding the use of emergency access.