In June 1979 Jimmy Carter's pollster, Patrick Caddell, wrote the president a devastating memo. Inflation was at 11%. Unemployment, the price of oil and tension with Iran was rising. Caddell was anxious. "For the first time, we actually got numbers where people no longer believed that the future of America was going to be as good as it was now," he said. "And that really shook me, because it was so at odds with the American character."

Against the advice of many, Carter tackled these concerns in a national address. He spoke of a "crisis of confidence" posing a "fundamental threat to American democracy". "[Confidence] is the idea which founded our nation and has guided our development as a people," he said. "We've always believed in something called progress. We've always had a faith that the days of our children would be better than our own." It was one of the most pilloried speeches in recent presidential history. Eighteen months later, Carter was out of a job.

Optimism is supposed to be one of America's truly renewable resources. Mythmakers injected it into the nation's bloodstream, inoculating the populace from a more lucid assessment of harsher realities. According to Gallup, every year since 1977 an overwhelming majority of Americans have assumed that they will be personally better off the following year, even when the previous year suggested the prediction was unfounded.

To question this confidence, even if the aim is to restore it (as Carter's was), is to invite electoral suicide. Voters generally want leaders to reflect the enduring nature of their myths rather than reflect upon their fragility. But to ignore that frailty can be no less fatal. In times of national crisis, voters want someone who addresses their underlying anxieties – not someone who covers them in rhetorical crazy-paving.

Now is one of those times, and this is one of those crises. A recent report revealed that between 2007 and 2010 the median American family lost a generation of wealth. Polls reflect the effect this has had on the national mood. Most Americans believe it unlikely that young people will have a better life than their parents – the highest number on record. More than a third think the country's best days are behind it. According to Gallup, the last time most Americans thought the country was heading in the right direction was January 2004 – it lasted a week. In short, a significant number of Americans feel that they are going backwards.

Financially, at least, they're right. Their wages have stagnated while the cost of living has skyrocketed. Since 1973, wages for 90% of American families have been flat, rising just 10% over almost four decades. Meanwhile, since 2001, a family's average health premiums have increased 113%; between 2000 and 2010, the cost of college tuition has increased by 32%; since 2006 median house prices have slumped 20%; a record 53% worry they won't have enough money for retirement. Those who can work longer, save more, consume less and expect less are treading water. The rest are sinking.

The central challenge of the forthcoming presidential election will be to provide a plausible narrative for how the country got here and how it will get out, without ever admitting the narrative exists or taking any responsibility for it. The central tragedy of the election is that neither candidate can plausibly do that.

The Republican nominee, Mitt Romney, is appealing to past glory, imperial power and manifest destiny. "There was a time – not so long ago – when each of us could walk a little taller and stand a little straighter," he told the National Rifle Association in April. "Because we had a gift that no one else in the world shared. We were Americans. That meant something different to each of us but it meant something special to all of us. We knew it without question. And so did the world. Those days are coming back. That's our destiny."

The trouble is, the reason Americans walk smaller is due to the legacy of the last Republican president. It was under George W Bush that the economy crashed, wars were lost, inequalities rose, deficits ballooned and global opinion of the US nosedived. Many more Americans still blame Bush for the state of the economy rather than Barack Obama; many more foreigners prefer the US under Obama than Bush.

Obama's message is more compelling but no less nostalgic. Referring to his grandparents in his last state of the union speech, he said: "[They] shared the optimism of a nation that triumphed over a depression and fascism … They [understood they] were contributing to a story of success that every American had a chance to share – the basic American promise that if you worked hard, you could do well enough to raise a family, own a home, send your kids to college, and put a little away for retirement. The defining issue of our time is how to keep that promise alive."

The trouble for him is that not enough people believe he can deliver on that promise. He started the year with flickers of economic recovery and the slogan: "America is back". The phrase was retired within a few months. People weren't buying it. Having run on a campaign of hope and change, too little has changed for too many and hopelessness has set in. For this he blames the Republicans, the Republicans blame him, and Americans blame all of them.

However, the main problem is not that American politics is so polarised that nothing can get done (though that is a problem), but that when they have reached consensus it has been for the worse. The repeal of the Glass-Steagall act, which deregulated the banks, and the North American Free Trade Agreement (Nafta), which sent jobs to Mexico where labour is cheaper and unions are weaker, both happened under Democratic presidents.

Indeed the only "people" with any cause to be truly optimistic are corporations (regarded by the supreme court as having the same rights as people), which are enjoying record profits and unprecedented political influence after the supreme court lifted limits on campaign contributions. The last few weeks have seen both Romney and Obama spend at least as much time persuading rich people for money as ordinary people for their vote.

After the 1904 election, the steel baron Henry Frick complained of Teddy Roosevelt: "He got down on his knees to us. We bought the son of a bitch and then he didn't stay bought." One thing Americans can be very confident about, is that no corporate leader would say that today.

Twitter: @garyyounge

• This article was amended on 18 June 2012. It originally stated that banks were deregulated by the Glass-Steagall act. This has now been corrected