The tube is in trouble. Not just this morning but every morning. The capital's most vital artery is both clogged by passengers and starved of the oxygen of investment. "The most important thing about the evolution of the subway as a symbol," writes Michael Brooks in Subway City, "is that it starts by expressing faith in the city's future and, once built, quickly becomes a handy rhetorical tool for expressing discontent with its present."

There are few places where this is truer than London. On its opening day, in 1863, "the crowd at the Farringdon Street station was as great as at the doors of a theatre on the first night of some popular performer", according to one observer. A few years later, Henry Mayhew overheard one workman say: "Just think of what these here trains save you at night after your work's over. If a man gets home tired after his day's labour he is inclined to be quarrelsome." While if he gets a ride home, and has a good rest "he is as pleasant a fellow again over his supper".

Today, as it ferries 1bn passengers around the capital every year, the tube is as liable to make a commuter quarrelsome as his work. It is a byword for underinvestment, overpopulation, industrial strife and political squabbling. In June, Ken Livingstone, the London mayor, will take the government to court over its plans to privately finance the underground. Next month, on May 3, on the day Tony Blair planned to call the election, tube workers will strike over safety and job security. The last time there was an industrial action, just over a month ago, the capital became one huge car park; paralysed as traffic that normally dawdles simply stood still. Journeys that would usually take 20 minutes were lasting two hours or more.

During his mayoral campaign, Livingstone once said that he would wake up in the middle of the night thinking of how the tube might be better run. It was a testament to his commitment to the city. Whatever you thought about his views on Irish politics or global capitalism, here was a man who was dedicated to London. And there was no greater, costlier or more cumbersome signifier for the city than the tube. At one stage, the battle for the mayor became a referendum on how the underground should be funded. But what started as a local transport matter has grown into a national issue - the first serious test both to the limits and possibilities of devolution and to Blair's Third Way. Commuters are also voters and none of the three main parties came close to Livingstone, who made his plans for the underground the centrepiece of his campaign. More than a dozen of Labour's most marginal seats are in constituencies where people often use the tube. Now Livingstone is not the only one for whom the tube is giving sleepless nights.

Back at platform three, Boyce is tackling a familiar problem. The Victoria Line northbound is a focal point for commuters from Sussex and suburban Kent who are on the way to the centre of town. Commuters, by their nature, are creatures of habit. They get to the station, come down the escalators, and wait for the train at the spot which will drop them at the most convenient place at the other end. Unfortunately, for all concerned, this means there is a huge bottleneck at one end of the platform. The other end, as the adjacent monitor indicates, is completely empty.

But as each individual ignores the request to move up the platform, the collective result is that a dangerous mass of bodies is crammed into one place. To avoid an accident the station assistants stop letting people through the barriers. Now there is a bottleneck in the concourse. Anxious people in three-quarter-length coats are fuming. The concourse fills up. Back at the platform the trains keep coming, but it will take several more before it is safe to let more people down. Meanwhile, the concourse fills up.

Boyce orders the entrance to the station to be closed off. "You don't like having to do it, particularly in winter when it's raining," he says. "But you have to for safety reasons. You can build the story backwards. It starts on the platform and ends up outside." With platform three clearing he gives the order to start letting them through again. "There's never a bottleneck for long. These people are late so they'll run down the escalator and it will all start again in 15 minutes," he says. On platform two, the westbound District and Circle Line, someone has pulled the passenger emergency chord. The signs that used to warn of a £100 fine for improper use have been removed now because of concern that people in genuine need might be deterred from signalling their distress because of the cost. A woman has fainted, and has been led off the train and assisted.

An abandoned duffel coat, masquerading as a potential security risk, has been found, kicked and cleared. It belonged to a tout - one of the men who try to sell tickets they have found to other passengers. "It's a shame he wasn't in it," says one of Boyce's colleagues. Back on platform three the bottlenecks have returned. Different faces etched with the same, time-starved angst held up by the same problem.

"I'll tell you the biggest change," says one manager who has worked on the tube for more than 30 years. "It's not so much the technology, although that is different. It's the people. They want to be everywhere yesterday. I don't ever remember people being in as much of a rush as they are nowadays. We have to provide the best service we can. But sometimes you want to say to them: 'It wasn't us who left the bag there.'"

Thanks to Britpop, Britart and Blair, in the latter part of the past decade London gained the adjective "swinging". For those who live there seething, straining or struggling would seem more appropriate. The pace of change is scarcely sustainable. London's population will grow 9.4% by 2021, far faster than anywhere else in the country. In 20 years there will be 7.71m in the capital - more than the current populations of Ireland, Latvia and Estonia combined, crammed into less than one hundredth of the space. "[London] is no longer simply the nation's capital, but now almost an international city-state," said a report by the Joseph Rowntree Foundation last year. Yet this massive and chaotic growth has, for the past 15 years, coincided with the absence of any democratic authority to develop or coordinate it. Since the abolition of the Greater London Council, London has literally been out of control. And one of the principal casualties has been the tube. On that much, probably, both Livingstone and the government can agree. Near Euston station a team of men staring at a wall full of grids are making sure the Victoria and Northern lines keep running smoothly. Sitting in a semi-circle monitoring traffic flows with the help of numbers and symbols the staff could be on the starship Enterprise. But the equipment they are using looks like it was rented from the Doctor Who archive of Jon Pertwee's era. "We are still working with 60s technology," says Peter Neal, services manager. "Everything you see around you where technology is concerned has been here since we started operating from this room. If you buy a car then you replace it every 10 years. At least. But train systems have to last for a lot longer."

The thorny issue is how to raise the money to set it right. The government favours a public-private partnership (PPP) whereby private companies operate the tunnels, tracks, signals and stations while the trains would remain in public control. Livingstone supports a fully publicly funded service, where the money is raised through bonds.

His overwhelming victory at the polls suggested that Livingstone had persuaded Londoners. With privatised railways fulfilling their political destiny as the poll tax on wheels, the Thatcherite orthodoxy that the private sector inevitably provides cheaper, better services than the state was in abeyance. But Livingstone could not convince the government. New Labour, it seemed, had attached itself to a dogma just as it was going out of fashion.

Livingstone vowed not to give up. He enlisted Bob Kiley, a former CIA agent, with an impressive reputation for turning around ailing underground infrastructures in America. He had done it in Boston during the 70s and then again in New York during the 80s. On paper, the socialist and the ex-spy made unlikely bedfellows. "I can't believe that I am trying to recruit a CIA agent," were Livingstone's first words to Kiley.

"Well, I find it hard to believe that I'm going to work for an unreconstructed Marxist," replied Kiley. The first tube ride Kiley took on his arrival saw him stuck in a tunnel on the Circle Line for 11 minutes. In practice their different world outlooks have caused problems. Livingstone supported the last strike. Kiley did not. "I told Ken that this strike was self-defeating," he said.

But the two things they did agree on was their support for public funding and opposition to PPP: "It has no basis in reality or common sense," says Kiley. Months of negotiations lead nowhere. If public opinion could have decided the outcome, Livingstone would have won. So skilful has his campaign been that the government has made the rare admission that even it has been outspun. "We took the view that through all the negotiations we would play a straight bat, not try and play to the gallery," said Prescott. "We are behind in the propaganda stakes but our argument is sound."

When months of negotiations collapsed and the government decided to push ahead with PPP, Livingstone decided to take it to court, claiming that the government had usurped his constitutional right to control transport policy and that its decisions will endanger lives. Mr Justice Sullivan said he was satisfied that the mayor's statutory powers were "novel, complex and untested in the courts" and that the issue was "of vital importance to Londoners generally".

Rattling around in the darkness, the promise of light in the cab of a tube driver is made long in advance. The next platform announces itself initially in a small shaft. Its growth is gradual, but little can prepare you for the glare of your arrival, where blank faces are waiting - a blur of people looking up from their books and newspapers. They have been waiting for you. Physical geography is meaningless here. You are travelling hundreds of miles a day and yet the city looks the same - long monotonous stretches of penumbra followed by short bursts of fluorescent yellow. Only through human geography can you map your way through time and place. Suits in the morning, tourists at lunchtime, schoolkids in the afternoon, impatient before 9am, rowdy after 9pm, blacker in the south and north, browner to the east. You are both in and out of control. Gliding through the city's underbelly, ferrying your human cargo to and from work, you are all-powerful. But on the Victoria Line at least, almost everything is operated electronically. With shifts lasting eight hours a day there is little to stimulate the mind. "Boredom is a big problem," says Jan Mowbray who works on the Victoria Line. "There's not much to do and it's dark all day. In the winter you leave home in the dark, work in the dark and then go home in the dark. In the summer it's really lovely and sunny outside and bloody dark in here. You can't win." And then she closes the doors and glides away. Through tunnels that offer occasional glimpses of light, but to which there is apparently no end.