Thanks for the detailed troubleshooting.
In the past there have been issues with slow smartcards and large private keys (2048bit and up)
What kind of cards, cardreaders and how large are your private keys?
I would also recommend to open a VMware Support Request since it does not seem to be a configuration issue and is probably something that needs to be looked at by our engineers.
// Linjo