So I asked my traffic manager at my traffic source and he told me that I will be charged even for those pop ups/unders which are blocked by ad blockers like AdBlock etc. Is there a way to avoid this since now the responsibility is on me (I used to think earlier that the traffic source would handle this) ?
No, you can not avoid this as long as you can't change the programming of AdBlock or the manner how your traffic source serves ads.
Which traffic source are we talking about? I believe pop up traffic from adware such as Trafficvance, LeadImpact, 50onRed can't be blocked even by ad blockers.