Among psychologists, the most interesting answer to the above question is that given by John Hibbing. He might be called the "rockstar" of the debate. He attracts attention because he goes down to the physiological and brain-science level for his evidence and conclusions. He says that what you believe is a product of what you are. He does not stress it but "what you are" is genetically determined. So he is looking for inherited physiological differences between Leftists and conservatives.
And he has made some progress. He has put people through a number of experimental tasks and found that the reactions he observes to the tasks do indeed differ as between the two ideological groups. He describes his findings as showing that "disgust sensitivity" is the key variable. Conservatives are more easily disgusted. Most generally, they have a "negativity bias", according to Hibbing. And last year he put up a big paper summarizing the evidence for his view. It is "Differences in negativity bias underlie variations in political ideology"
I have long argued that Left/Right differences are largely inborn so my critique of Hibbing is not to contest his findings but to question the "spin" he puts on them. You can find a pretty good summary of his experiments here and I think it is easy to see that what Hibbing calls "negativity bias" could just as well be described as caution -- and caution has long been said to be the essence of conservatism. So Hibbing has confirmed some old wisdom rather than telling us anything new.
Hibbing's big article was published in an open review journal so critiques of it keep multiplying. One such critique that I have noted recently was "Not so simple: The multidimensional nature and diverse origins of political ideology" by Stanley Feldman and Leonie Huddy. They make two points that I think are pretty right:
They say that "negativity bias" is characteristic of neurotics and that all the studies show that conservatives are not particularly neurotic. I observed that in my research too. So that is a bit of a stake in the heart for Hibbing. His "spin" on his results has undone him. If he had simply described conservatives as cautious, that criticism could not so easily be levelled at him.
Their second point is that there is no single Left-Right dimension. Economic conservatism and social conservatism are quite different. So Feldman & Huddy conclude that Hibbing's work is pretty useless because he has mixed up two different things. And it is indeed true that those two types of attitudes are very distinct factorially. I noted that in one of my papers long ago.
So they are right but I am prepared to defend Hibbing on that one. Although there are two distinctly different types of conservative attitude, they are not totally different. As I found, they do correlate, albeit weakly. And that is why the "big tent" of the GOP succeeds. The two types of conservative do find some things in common, a respect for the individual, mainly.
And as we see in "Political Attitudes Vary with Physiological Traits", Hibbing uses the Wilson scale in his research, which is primarily a measure of social conservatism. And I have shown elsewhere that social conservatism is the big one for separating people. Economic conservatism is arguably more important to our future but it doesn't get the blood boiling like social issues do: Abortion, homosexuality, religion, tradition etc.
So Hibbing may not be measuring overall conservatism but he is measuring social conservatism and that is the most central sort of conservatism. So I would summarize his findings as showing that social conservatives are instinctively more cautious than others. And I see no problem with that.
Hibbing uses "Negativity" rather than "caution" to describe conservatives because he wants to rubbish conservatives (though he says he does not). "Negative" sounds a lot sadder than "cautious". But in so doing he lands himself in trouble. I have noted the Feldman & Huddy comment on that but there is in fact a bigger vat of boiling oil he falls into:
As is noted here, who are the "negative" people when it comes to global warming? Warmists are almost entirely Leftists but it is they who are vastly negative about the climate and our future. They predict imminent catastrophe -- while conservatives are mostly just amused by the scare. Conservatives say in summary that: "global warming is not a crisis, the likely benefits of man-made global warming exceed the likely costs, and mankind is not the scourge on Earth that liberals make us out to be"
And again, referring to conservatives simply as cautious would not enable that criticism. Warmists do say that they are the cautious ones but to swallow the arrant nonsense that is global warming would have to be a height of incautiousness. Conservatives just look at the evidence and see that there is no need for caution in the matter. Here's a graph of the amount of global warming we have had in the last 18 years -- none:
So two cheers for Hibbing. He has drawn attention to the biological basis of ideology but he should stop stretching the implications of his findings in a Leftist direction. He just makes a fool of himself with that stretch. He was pretty reasonable -- even humble -- in a 2012 paper. He should try more of that.