The main vulnerabilities of a smart card are the loss of a card and PIN. Make sure that you have a clear policy on lost and stolen cards, and that end users notify you of a lost card so you can revoke the end user certificate.
The Admin Key is also a vulnerability. With the Admin Key someone in physical possession of the card can reset the PIN and take control of the credentials. Make sure the Admin Key is stored securely by using a CMS, or by printing the Admin Key out, or storing it on a USB token. Do not store the Admin Key on a networked system. Alternatively consider randomizing the Admin key for each card, and not storing the Admin Key at all.