yes, of course you can remove it after installation.
However, if you will need to add nodes later, you will have to reestablish keys again.
What if you just rename authorized_keys file to something else?
This way you preserve all the settings and revoke passwordless assess from the cluster.
And it could be reestablished in a matter of seconds…
Just an idea….