Pity George Bush. Scanning eight years of calamity for the lowest point in his presidency could not have been easy. Among the top contenders: Abu Ghraib; failing to act on threats of an al-Qaida attack before 9/11 or find WMD in Iraq; helping to collapse the economy; being forced to withdraw a supreme court judge choice; and failing on immigration reform.

But no. According to his recent memoir, the nadir came when Kanye West, a black singer, accused him of racial neglect in the aftermath of Hurricane Katrina. West said: "America is set up to help the poor, the black people, the less well off as slow as possible … George Bush doesn't care about black people."

Given what was happening at that time, this hardly seemed outrageous. With bodies floating in the street and people stranded on highways, Bush's director of the Federal Emergency management Agency, Michael Brown, said of the mostly black crowd that had gathered at the convention centre: "We're seeing people that we didn't know exist."

But for Bush, West's remarks went beyond the pale. In an interview this month, he said: "It's one thing to say, 'I don't appreciate the way he's handled his business.' It's another thing to say, 'This man's a racist.' I resent it. It's not true."

There are many issues relating to Bush's pique, but let's just concentrate on two. First, his umbrage at an accusation that had not been made. West did not call him "a racist". To accuse someone of not caring about something is not the same thing as accusing them of discriminating against it. West has a good command of the English language. Had he wanted to call Bush a racist he could have. Bush's inference was by no means absurd; but it was his to make.

Second, the fact Bush decided to respond in this way tells us a great deal about the passive-aggressive nature of modern racial discourse. For we have moved to a place where accusations of racism, real or imagined, are routinely understood to be more egregious than actual racist acts themselves. As a means of avoiding conversations about what they have done, people instead insist on what they are not.

In this case, Bush is more upset by the claim that he didn't care about black people than the fact that a disproportionate number of black people died during Katrina unnecessarily because his administration did not care enough to save them.

Debates about what is motivating the rightwing resurgence against Obama's presidency often take a similar course. In the many conversations I have had with the right, I have not once even inferred they might be racist. But the retort that they are not racist comes back just as sharply as if I did. So let's start by pointing out that American conservatives have plenty of reasons to oppose Obama that have nothing to do with his race. For all his shortcomings he remains the most progressive president for at least 60 years. He has expanded public spending and healthcare; drawn down troops from Iraq; and campaigned on redistributing wealth by raising taxes on the rich. Bill Clinton was nowhere near as liberal – and look what they did to him.

Nor is racially charged rhetoric a preserve of American conservatives. During the democratic primary campaign, Hillary Clinton's chief strategist, Mark Penn, argued that Obama should be undermined on grounds of race. "His roots to basic American values and culture are at best limited," Penn said. "Let's explicitly own 'American' in our programmes, the speeches and the values. He doesn't."

To ask where racism ends and politics begins sets up a false dichotomy – US politics has always been steeped in race, and racism has always been a political and electoral force. The psychic scars of centuries are not removed as a result of one person being elected. Indeed, if the racial polarisation of the electorate in the mid-terms is anything to go by, they may have deepened and been made even more raw as a result of it.

Let's also concede that his race is a factor. It would be remarkable if it were not. The reason his election had such symbolic resonance was precisely because it was assumed so unlikely in a country where black people are overrepresented in jail and among the poor, and underrepresented in politics and power. Since the 1960s, American conservatism's national electoral strategy has hinged, in no small part, on leveraging white southerners from Democrats with scarcely veiled racial messages.

Attempts to deny that Obama was born in the US and that he is Christian (common among Republicans and predominant among Tea Partiers) are, to some degree, proxies for race. They are a way of casting him as "other" without touching less acceptable bigotry. A recent Washington Post survey of Tea Party groups found that 11% said Obama's race, religion or ethnic background were "very important" or "somewhat important" in the support their group has received. A relatively small number of racist posters have consistently been seen at Tea Party rallies.

So while racism may significantly shape the character and inform the intensity of opposition to Obama (the week he was elected gun sales rose 50% compared with the previous year), it does not drive it. But his particular constellation of identities are better understood not so much as objects of racial animus but as signifiers for a far broader set of geopolitical, economic and demographic anxieties.

For the poorest 90% of US families – the overwhelming majority of whom are white – median income has been effectively stagnant for a generation. Meanwhile social mobility has stalled. In this situation, many white Americans do not sense their experience compared with non-white Americans is one of relative privilege – because over the last 30 years, they are relatively no better off.

Many blame this on the outside world. From 47 countries polled by Pew in 2007, Americans showed the sharpest decline in their support for foreign trade and had the least positive view of it. The US may have been one of the principal motors of neoliberal globalisation, but its citizens are also its victims. In the absence of any vehicle for international class solidarity, threats of outsourcing, product dumping, deflating the dollar and Chinese creditors provide the material basis for a strain of xenophobia that goes beyond a simple loathing of foreigners.

To the sting of economic vulnerability is added the indignity of geopolitical decline. As the sole global superpower, the US would once have been able to rig the competition with carrots, sticks and, if need be, B52s. Now it must accept that Indians, Chinese, Brazilians and others can also change the rules.

Add to this failed wars against predominantly Muslim countries after terrorist attacks by Islamic fundamentalists, a broken immigration system, and projections that non-whites will be a majority by 2042, and you have the roots of a race-based backlash. Put bluntly, being American is no longer what it used to be – at home or abroad. And for those particularly invested in the relative privilege of being a white American it is not difficult to see how the election of a black president – with an African name and a foreign father who was a non-practising Muslim – could become a focus for discontent.

Race was too narrow a lens through which to examine opposition to Obama, as it was to understand what happened in Katrina. Racism cannot explain all of it. Indeed it's not even the half of it.