To delete a credential (certificate and keys) that is stored on the PIVKey you can use a utility, such as vSEC-CMS, but you can also use Certutil.
To do so you need to know the exact container of the credential you want to delete. You can find this out by running certutil:
certutil -key -csp "Microsoft Base Smart Card Crypto Provider"
Make sure you identify the correct container name. Note the name of the container may contain the certificate template name.
To delete the container and its associated certificate run:
certutil -delkey -csp "Microsoft Base Smart Card Crypto Provider" [container-name in quotes]
Note in the example above, the container that is deleted is the default PIVKey Credential for a PIVKey C910 card.