When the police asked about setting up a stall for Scarborough's first ever gay pride event last September Christine Mark turned them down. She feared the turnout might not exceed six – her, her partner of 33 years, Jane, and two other couples. There had been some reticence in the community when she was planning it.

"Some people had asked whether we might not have an indoor event where they would feel less exposed. I told them they misunderstood the idea of what Pride was all about."

In the end they needn't have worried. More than 100 people showed to up to the picnic in Peasholm Park. Some came from as far away as Sheffield, but most were local. The staff at the park dressed in pink and baked a pink cake. More than 30 years earlier the mayor in this charming north-eastern seaside town of 50,000 residents had refused to host a gay rights conference, on the grounds that he didn't want "that kind" in the town. This time the mayor turned up with his wife and cut the cake.

There was one letter of complaint about the event in the local paper, but the next week there were several more of support, plus an editorial endorsement. After Jane and Christine tied the knot in a civil partnership the most common reaction was: "Why did it take you so long?" When they pointed out to people that such unions used to be illegal, people looked surprised.

Whatever happens on Thursday there are all sorts of ways in which Labour has left an indelible mark on how British people view themselves and each other. The overall record is contradictory. Broadly speaking, Britain feels like a more open-minded, cosmopolitan and less parochial place than it was 13 years ago.

The country goes into this election with more than double the proportion of women MPs and treble the number of minority MPs than there were before Labour came to power. In 1997 Ben Bradshaw was the lone gay parliamentarian. On Thursday the Tories alone are standing 20 openly gay and 44 minority ethnic candidates, while a record proportion of all the candidates standing are women.

But the results tell us something quite different. The election of more women – still low by international standards – did little to break the blokeish mould at the helm. Meanwhile, in last year's European elections the BNP won two seats and more than 6% of the vote. Ukip came second, ahead of Labour. The BNP and Ukip got a quarter of the popular vote between them. In the five years after I left the country for America in 2003 racist attacks rose by 28%. Islamaphobia and antisemitism have seen a steep rise. Few would have predicted any of this when Labour came to power in 1997.

It's not difficult to see where some of this has come from. Without doubt, the biggest and most dramatic change in recent years has been the arrival of eastern European migrants.

In Stevenage, where my mother used to have to order West Indian hot pepper sauce a week in advance, there is now at least one Polish delicatessen in the town centre. At my local Nandos in Hackney, north London, which used to be staffed primarily by black students of west African origin, I was now being served by Katja from Latvia; I saw no black faces at the grill. I found these changes quite exciting, but I did wonder how those other students were funding their way through college. After a while I stopped assuming that white people in restaurants and hotels necessarily spoke English as a first language.

This held two main consequences for British identity. First it started a new chapter in our immigration narrative, away from our postwar, post-colonial past and towards a more pan-European future. Second, it finally ruptured the connection in the public psyche between immigration and race.

This was what was most illuminating about Gillian Duffy's comments to Gordon Brown. She lives in a town where one in five is Muslim and the Bangladeshi population increased 58% in a decade. But when it comes to immigration her specific concerns were about eastern Europeans.

Similarly at the debate in Birmingham on Thursday it was a black audience member that voiced concerns about immigration.

So the movement of people and the colour of people are no longer routinely confused. The assumptions about who is local and who is foreign are no longer assumed to be discernible by the naked eye. Thanks to war and terror the emphasis has also shifted away from racial difference to religious and cultural difference in particular.

None of this means racism has disappeared. Xenophobia, religious intolerance and racism are not synonymous but bleed into one another. The current framing of the debate about immigration has clear racial and racist overtones, even if it is being driven in part by an anxiety about white foreigners. But the faultlines have clearly shifted from colour to culture, race to religion, language and ethnicity.

It goes without saying that Britain would have undergone profound cultural changes in more than a decade regardless. Thanks to new technology, easier communication, cheap air fares, demographic shifts, EU expansion, war, terrorism and globalisation, the world, and our place in it, has been radically transformed.

Nonetheless, it is worthwhile remembering how far things have come. Back in 1997 Bradshaw's Conservative opponent, Adrian Rogers, openly branded his Labour counterpart "Bent Ben" while describing gay love as "sterile, disease-ridden and God-forsaken".

A year later, not long after Peter Mandelson and Nick Brown were outed, the tabloids sensed an opportunity to spark a moral panic.

The Sun asked: "Tell us Tony: Are we being run by a gay mafia?" The Daily Mirror claimed the public had a right to know their politicians' sexual orientation and ran a phone poll asking readers if they agreed. The next day they had a shock. Sixty-two per cent of their readers voted against them.

A few days later, after a considerable backlash, the Sun issued what amounted to an apology. "The Sun is no longer in the business of destroying closet gays' lives by 'exposing' them as homosexuals," it said.

Polls today show a majority of Britons support gay marriage, including a majority in every party. Unlike in the US, where support is significantly lower and the issue has become contentious, here it seems almost banal. I recall seeing accounts of Elton John's ceremony where the principal talking point was his outfit.

So Labour may have been pushing at an open door, but others were pushing back. It repealed Section 28, equalised the age of consent and introduced civil partnerships. Attitudes may have shifted, but they did not shift by themselves. "If I were voting on gay rights alone I would vote Labour," says Jane, who will be voting Green.

Nor should this suggest that these developments have been universally accepted. Last week a Tory candidate was forced to resign after saying that while he "supports the rights of homosexuals to be treated within concepts of (common sense) equality and respect, he will not accept their behaviour is normal". Over the weekend it was revealed that Philippa Stroud, a Tory candidate and influential force within the party on social policy, had set up a church to "cure" gays by driving out their "demons". But it is difficult to imagine that these advances could be reversed.

In all of this, gay rights really serves as a metaphor for the sweeping changes that have taken place and the anxieties they have produced. Rogers railed not just against what Bradshaw stood for, but who he was: "He's a media man, a homosexual, he likes Europe, he studied German, he lived in Berlin, he lives in London and he rides a bike. He's everything about society which is wrong."

Shortly before I moved to the US it seemed as though the conduit for these prejudices was the Countryside Alliance, which held the biggest demonstration in British history at the time. Galvanised by the ban on foxhunting, many in their ranks sought to use rural as a proxy for traditional, and urban as a byword for decadent.

At one demonstration a poster bearing the Alliance's logo depicted the agriculture minister, Nick Brown, as a man "who loves gays and buggers the countryside". The Earth Dog, Running Dog magazine – which the carried the logo, even though the Alliance distanced itself from it, described the black former MP for Bethnal Green and Bow, Oona King, as "typical of her species", and told her to "direct her talents to advising her scrounging supporters on how to claim more handouts".

At the time, before the anti-war movement supplanted it, the Countryside Alliance looked like it could present itself as the largest immovable object to the New Labour government. Where all that rural resentment and energy went I'm not sure.

What is certain is that while many of the progressive changes feel permanent they also seem, to many, quite precarious. Shortly after the publication of the Macpherson report there was a brief moment where racial and ethnic difference was openly embraced.

That period saw Chris Offili and Steve McQueen win the Turner Prize, Goodness Gracious Me make its TV debut and the publication of Zadie Smith's White Teeth. Then came 9/11, Afghanistan, the Iraq war, the war on terror, 7/7 and the onset of moral panic. The day London was awarded the Olympics, our multicultural society was our selling point; the day after 7/7 it became, in some eyes, our greatest liability.

Similarly, for all their achievements Jane and Christine have been ignored by gay friends because they were with families or friends and didn't want to be exposed. After the picnic they also threw a masked ball so that those fearful of being exposed could still participate. "We're moving in the right direction but there's a long way to go," says Christine.

Across town at the Mecca bingo hall most feel comfortable with lesbian and gay equality. "That's the way it is nowadays," said one elderly man. "You've got to live and let live."

Only one said homosexuality wasn't "normal", but added "It's different though for younger folk."