Early in the morning on Sunday 23 July 1967 the Detroit police raided an after-hours drinking establishment where more than 80 black men and women were celebrating the return of two Vietnam veterans. This in itself was hardly rare. Police used to raid "blind pigs" all the time.

What was extraordinary was what came next: an outpouring of protest, violence, looting, police brutality and, ultimately, full-scale federal military intervention. "To live in America, until recently, meant to be far from war," says Calliope Stephanides, the protagonist in Jeffrey Eugenides's novel Middlesex, which is set in the city at the time. "Wars happened in south-east Asian jungles … But then why, peeking out of the dormer window, did I see a tank rolling by our front lawn?"

Before the week was out there were 43 dead, 467 injured, more than 7,200 arrests, and more than 2,000 buildings destroyed: the deadliest riots in US history.

Nathan Caplan, a psychology professor at the University of Michigan's institute for social research (ISR), was unconvinced by his colleagues' claims that the riots were simply an expression of immaturity and social deviancy. So he went to the affected neighbourhoods with one of his graduate students to conduct some field research.

"There was still smoke in the air and bullets flying when we got there," recalls Caplan. "My academic colleagues had a habit of interpreting reality as though it's just a special case within theory. God forbid that anything they did became useful or that they actually spoke to anybody."

At the same time, Philip Meyer, a national correspondent for Knight Newspapers, which owned the Detroit Free Press, was also trying to break out of the constraints of his discipline. Just a few weeks earlier Meyer had returned from a career break as a Nieman fellow at Harvard, where he had studied social research methods. "Politicians were using them and I felt journalists had to understand them too," he explains.

When the riots in Detroit started he saw an opportunity to apply academic rigour to journalistic reportage. "It struck me like a bolt of lightning," says Meyer. "The University of California had just released their report into the Watts riots and I thought, we could do this too. But people wouldn't have to wait two years for the results. With journalists we could do it faster."

And so they did. Caplan and Meyer got together, drafted a questionnaire and then trained 30 black interviewers to go into the affected areas and gather information. The next week the interviewers went into the field, sending each day's interviews back to the university. In the third week Meyer and Caplan analysed the data and Meyer started writing it up. Just a month after the disturbances had started, the Free Press published its findings.

Entitled The People Beyond 12th street: A Survey of Attitudes of Detroit Negroes After the Riot of 1967, the report's methodology and language are very much of their time.

Seeking to explain black Detroit to white America, it reads like an anthropological survey of a foreign land. "In their alienation the rioters display some similarity to hippies," reads one part. "Both feel that the world is wrong and they want to set themselves apart from it. But hippies accept their share of the world's guilt while rioters project it. The hippie hands you a flower and says: 'Peace.' The rioter shouts: 'Get whitey' and throws a rock."

Nonetheless, even though by modern standards the sample was small and only a minority of those they spoke to were rioters, its data was revelatory.

It showed that, contrary to popular belief, there was no correlation between economic status or educational levels and propensity to riot. Nor had the riots been the work of recent immigrants from the south. The main grievances were police brutality, overcrowded living conditions, poor housing and lack of jobs.

There are relatively few similarities between the Detroit riots in 1967 and those in England last month. The unrest in Detroit was part of a national conflagration led largely by the young. A riot had ended in Newark a week before and would continue in Milwaukee and Washington DC just days after. But these disturbances were rooted in racial conflict. "Detroit was becoming increasingly black and the police force and the city government remained primarily white," explains the veteran Detroit activist and author Grace Lee Boggs. "So the institutions in the city were seen as occupation forces."

It was also a pivotal moment in American racial history. A period of mass activism in the south had produced civil rights legislation just a few years before, securing black Americans the right to equality and the vote. 1967 was the year Muhammad Ali refused to serve in Vietnam, Cleveland became the first major city to elect a black mayor and bans on mixed-race marriages were finally made unconstitutional.

But while the contexts in England in 2011 and in Detroit more than 40 years earlier were quite different, many of the most popular explanations for the root causes were startlingly similar. The youth of those involved and the looting led many to dismiss those involved as just riffraff running amok.

"I was hearing that the riots were the product of immaturity and social deviancy by the lumpenproletariat," recalls Caplan. "But that's not what we were finding."

Then there is the apparent disconnect between the event that triggered the riot and the prevailing conditions that might explain it. "Every riot has a precipitating event," explains Meyer. "But then there are always more subtle factors."

Indeed, thanks largely to union jobs at the city's car plants, African Americans were materially better off in Detroit than elsewhere in the country. More than 40% owned their own homes. The city had the largest chapter of the NAACP, the nation's oldest and largest civil rights organisation, in the country and was the only city with two black congressmen.

By revealing there was no link between income or education levels and rioters, and that those born in Detroit were three times more likely to have been involved than those from the south, the ISR/Detroit Free Press survey provided evidence that effectively countered popular misconceptions. "The riffraff theory and the southern migrant theory just stopped being propagated," says Meyer. "They were more likely to be young but that's because they had less to lose. In that sense the young were like leading edge consumers."

"It gave an empirical basis to address some interpretations and it put them straight into the public sphere," says Caplan.

It also gave voice to the complex attitudes towards social unrest in areas most likely to play host to them. The overwhelming majority of black people in Detroit at the time (84%) thought the riots could happen again. Only a quarter believed they had more to gain than lose by resorting to violence. But, explained Meyer at the time, "even those who think the long-term effects of violence are likely to be bad see some compensating benefits". "They know we mean business now," one 31-year-old told him.

The Kerner commission, set up by the president, Lyndon Johnson, to examine the cause of the riots in Detroit and elsewhere, used the data from the Detroit Free Press/ISR survey in reaching its findings. "What white Americans have never fully understood – but what the Negro can never forget – is that white society is deeply implicated in the ghetto," it argued. "White institutions created it, white institutions maintain it, and white society condones it." Famously it concluded: "Our nation is moving towards two societies, one black, one white – separate and unequal."

You can certainly see that society in Detroit today. Most respondents to the survey believed the attitudes of whites towards blacks would improve over the following five years. In fact whites just left the city en masse for the suburbs; a few decades later more affluent blacks followed. Following the collapse of the US car industry a couple of decades later, Detroit might best be described as post-industrial. But the immediate impression is post-apocalyptic: empty lots, abandoned houses, multistorey buildings left derelict, entire blocks boarded up. Detroit has lost 25% of its population in the past 10 years. It's what you would imagine a city would look like when capitalism decides that it doesn't really need the people any more.

Five minutes from 12th Street (which has been renamed Rosa Parks Boulevard) and Clairmount, the intersection where the riot started, you can buy a house for $1200 (£750). The only ad on the side of a row of dilapidated storefronts is for criminal defence lawyers. And even that has been defaced.

So if poverty and poor education did not cause the riots, what did? Alongside police brutality, Boggs describes an alienated youth who saw that technological changes in manufacturing would make them economically obsolete. "Young people felt expendable. They had to make a way out of no way."

The former black panther Wayne Curtis suggests that the apparent wealth belied lives of discrimination. "Growing up we saw people riding around in Cadillac convertibles. But we didn't see the conditions they worked in and the racism in the factories. They still had to deal with the police. We just didn't understand how marginalised they were."

Curtis recalls thinking that the civil rights victories had settled the question of equality once and for all. "We were told that this was it; that if we could vote everything would be OK. But it took a while to understand how the power structure works."

Both Caplan and Meyer call this the problem of rising expectations. Having fought for and ostensibly won social equality African Americans found that did not translate into economic justice. "Young black kids just weren't getting jobs," says Caplan. "The opportunity structure was just too narrow and everyone knew it. So you would see young people breaking windows and looting stores. But it was the older ones who were more frustrated and more militant in their views."

• This article was amended on 8 September 2011 to add a link to an article that was part of the source material for this feature.