James Watson was an idiot because he based his claims on Bell Curve

The issue is not with the fact that you might label him an idiot or not.
The issue is that he made a statement and suddenly was uninvited from lots of conferences and "retired". Stupid statement = massive consequences.

This is a lot about externalities:

Now forget about Watson a bit and think on any person that might have something interesting to contribute about the said topic (cognitive differences between human populations), that person might be so inclined - because of perceived consequences - to avoid addressing the issue. Even if the person decides to go forward others might be inclined to ignore the person (say, avoid publishing the opinion).

With time, people will naturally and unconsciously avoid those issues (might hurt their career, reputation, social life, ...).

There should be some tolerance for opinions that are way off the mark. Calling him an idiot, that is OK. "Retiring" and uninviting a person because of them, is too much. Not only because of the said person, but because of the consequences to us all, and the perceived freedom that we have.

PS (regarding Watson) - Irrespective of stupid comments and most probably underlying crude racism - and classism [1] - I do think that opinions like these are worthwhile considering. Unfortunately this opinion is voiced by a person that seems to be, simultaneously, a crude racist.

[1] People are so preoccupied in pointing that the guy is a racist that they seem to not notice that he is also very classist.