Husband's 'special friend' isn't his wife
The most "special" people in your husband's life should be you and your kids. After that, other friendships he has should be shared with you.
It is natural for your husband to accuse you of blowing this out of proportion -- but even if you are, what is he prepared to do to rebalance the proportion? He needs to introduce this special person to you, and then he needs to be sensitive to your anxiety and respectful about your reaction.
LIers missing in Nepal ... Dolly Parton light show... LI's Greenway project... Dom's Famous Chicken Fingers