We had a similar problem in our block recently.
When we confiscated the key safe from its common property “hide-away”, the tenants in question superglued the building door locks open. We subsequently tracked the “visitors” to a number of units.
We put up notices stating that anyone entering the building without a security key, unless admitted via the security intercom, would be reported to the police and charged with trespass. This reduced the turnover until we could get the offending tenants evicted.