You’re right that the committee can’t just cancel swipe cards, at least not legally, but they can restrict swipe cards to registered residents.
The law demands that the owners corporation be informed of any changes to the lease of a lot, with information provided as to the identity of the new resident. This is routinely ignored by holiday letting hosts who want their guests to have the same rights of access as residential tenants, but don’t want to tell us who they are.
Short-term letting agencies expect us to stick to the letter of the law when it suits them and ignore the law, such as the terms of development approvals (namely, no holiday lets) when it doesn’t.
So I would make it a security issue. Announce that there will be a periodic audit of the swipe cards for security reasons that may result in swipe cards being cancelled when they are held by someone other than the registered leaseholder or their registered sub-tenant.
Some buildings even have a system that matches the face registered on the computer with the face of the person trying to gain access. Others use fingerprint technology, especially to restrict access to facilities.
As for the threat of legal action – personally, I would push it to the limit (mediation) then back down and find another way of frustrating these parasites.