Who, exactly, are we talking about when we discuss people locked up in places like Yarl's Wood immigration removal centre?

They are people whose applications to remain in this country have been declined, people going through the tortuous process of appeals, who are fighting desperately to stay in a place where they believe they might have a chance of a better life.

Perhaps they have lived in the UK for years. They might have started to put down roots. They might belong to a church, even.

At a legal level, though, they are still strangers in a strange land.

They might be failed asylum seekers, with stories to tell of terrible oppression and brutality. They are afraid and vulnerable. However, they have not told their stories well enough to have been believed, or they have come from countries which don't officially persecute people, so their applications have been refused.

Or they might have migrated for 'economic reasons', like the desperate people trying to make it across the Channel from Calais. Of course it's true that the vast majority of these people want to work and pay taxes rather than sponging off the state. No matter: they can't come in.

However: as the parliamentarians at the launch of yesterday's report stressed, it's possible to have very different views about the scale and desirability of immigration and still come to one mind about the workings of the system designed to control it. For good reasons or bad, these people don't belong, and because of that they can be detained at Her Majesty's pleasure. Only it isn't put like that: detention is not technically imprisonment. And if in this year of celebrating the 800th anniversary of Magna Carta, which brought into the laws of England the principle of habeas corpus, that distinction without a difference rings even more hollow, that's just another example of hopelessness and history rhyming.

The sad thing is that it doesn't have to be this way. Britain is unusually harsh in its treatment of people it wants to get rid of – and unusually inefficient. Sweden, for instance, has three times the number of asylum seekers as Britain and only a tenth of Britain's population. It detains only a few hundred, rather than the thousands kept at Yarl's Wood and other centres.

Let's be honest: many, probably most, of those who work at places like Yarl's Wood are decent people who try to treat detainees fairly and would have been horrified to see Channel 4's documentary last night. They don't want people to be hurt of victimised. They'd be appalled by some of the treatment that was depicted.

But there's a fundamental problem with our immigration system that is exposed by the parliamentary report into detention published today. We treat detention not as a last resort, as the Home Office guidelines originally envisaged, but as a standard procedure. The result is that people are trapped in a system that doesn't provide any incentive for deciding their cases quickly and fairly. So justice is delayed and therefore denied.

Furthermore, there is something corrupting about fences and alarms: not to the inmate, but to their guards. Once the world is divided between those on one side of the bars and those on the other, those on the outside become prey to the severest temptation. It becomes possible to treat the captives differently. They are not us, after all. They belong to a different tribe. They are different, dangerous, contemptible and guilty.

Those who lead institutions like Yarl's Wood are responsible for setting a moral tone which guards the guards against this dark distinction. But the parliamentary report shows that at the root of the problem is a philosophical failure. Instead of treating people with respect from the outset, they are regarded as liars. They are routinely disbelieved, harried, not represented, transferred between institutions, left unable to maintain their social connections. Of course they are locked up. Why wouldn't they be? In some twisted way they have become the enemy.

Partly this is possible because of the way asylum seekers are demonised by certain sections of the media. Partly it is because no one has grasped the scale of the problem or the coherence and simplicity of the alternatives.

The latter, at least, is no longer an excuse. It is time for a change. A few years ago the detention of children was outlawed after public pressure mounted. Now it is time – long past time – to make sure that no one is detained at all who does not need to be.

Follow @RevMarkWoods on Twitter.