Let 'em wonder. Seriously though, wearing a man bag is supposed to be ambiguously sexual. If you think you are going to send 110% hetero beams in every direction while sporting a man bag you are delusional.
Women love mystery. They spend billions of dollars a year on chick flicks and romance novels where the ideal man can't be figured out right away. It might not be acceptable fashion according to whoever writes the rules but it will pique the interest of women you want to date.
Also finally, fashion is about expressing yourself. If you want to do it, fucking do it.