Another way of thinking about the Summers male/female brain debate: someone somewhere in the past few days mentioned the ratio of male to female prisoners in this country. As of 2003, a man was fifteen times more likely to be incarcerated than a woman. For anyone contesting simply the possibility that men might be, on average, slightly more likely to be brilliant mathematicians than women, think about those different incarceration rates. The most likely explanation of this 15-to-1 ratio is that men's brains are more prone to violent behavior than women's -- or at least that there is among men a wider statistical distribution of violent/nonviolent attitudes, which results in more unusually violent men, and thus more prisoners. So if we accept that a significant difference exists in the way that each sex deals with violence, why shouldn't there be the possibility of a much smaller difference in the processing of math?
Of course, if you think that this 15-to-1 ratio of male prisoners to female ones is purely the work of social bias, then I think you have to ask: who, then, are the primary victims of sexism in this culture? Sure, it's a shame that women aren't making as much on the dollar as men, and aren't getting tenure often enough. But if society is, for totally arbitrary reasons, putting fifteen times as many men in prison as it is women, I think that's a more grievous offense.
Obviously, there's a common answer in both the academic context and the penal one: both genetic and social factors are responsible for the disparity. Men start with a stronger tendency towards violence and society amplifies that tendency over time. But the fact that more men are in prison isn't by itself a sign that we live in a society that discriminates against men. The same logic applies for the number of tenured mathematicians at Harvard.
P.S. And don't bother objecting that, by this logic, the high rates of African-Americans in prisons means that there's an innate capacity for violence among that group. The genetic differences between the sexes are hundreds of times more significant than the differences between the races. You can't look an fMRI of someone and say, "That's an African-American brain, or a Caucasian brain." But you can differentiate between a male and female brain.