Lidi was mostly right but what I feel is important to emphasize here is that natural selection doesn't work on one sex alone. If it did, you would inevitably see animals that looked morphologically very different from one another, as though they were ACTUALLY members of another species. This CAN happen, but only does when extreme sexual dimorphism becomes an advantage.
You have to think of male and female members as being part of a larger gene pool, one from which both sexes draw. Natural selection will alter that gene pool over time, making additions and subtractions, paving over certain spots and enhancing others. In fact, many of the genes men and women share contribute to sexual differences in the presence of different hormone levels. It's much more difficult for natural selection to act on these genes in a way that affects both sexes positively.
Female mammals have nipples because they serve such an obvious purpose...but there is no similarly negative influence on male mammals to NOT have them. On the flip side, if a man was born WITHOUT nipples, that might be just fine, but if his daughters, in turn, were also born without them, that unique genetic code might be weeded out, over time.