It's the most well-known psalm and one of the most well-loved bits of the Bible. It's been used in various films including The Wicker Man, The Elephant Man, Titanic and War Horse. It's been recorded by artists from Pink Floyd to U2 to Jay Z. It's the best selling, smash hit, guaranteed-to-get-them-on-the-dance-floor psalm.

The purpose of psalms, broadly speaking, is to express things about God and to God. Some psalms are good for when you're feeling great. Some psalms are good for when you're feeling down. Some psalms are loud and celebratory, some psalms are mournful and more suited to a funeral.

The reason Psalm 23 has remained so popular and profound for believers around the world throughout the ages is that it doesn't just work in one given situation. It works in any given situation.

King David, who wrote the Psalm, uses the image of God as a shepherd and himself as a sheep. This is an image Jesus himself uses and is used about Jesus, and would have resonated very closely with both those reading David's psalm and also those listening to Jesus. They were part of a farming society and understood how important it was for the shepherd to look after and guide his sheep – in fact David had been a shepherd before becoming King.

The Psalm takes us on a journey with God leading David.

The Lord is my shepherd, I lack nothing.
He makes me lie down in green pastures,
he leads me beside quiet waters,
he refreshes my soul.
He guides me along the right paths
for his name's sake.

The shepherd is gently guiding the sheep, giving him rest, leading him beside quiet waters and making sure he goes along the right path.

This is a psalm that speaks to us when life is good. It tells us that we can be sure that it's good because God has made us, he loves us and he also made the world and everything in it. When times are good we can happily say this psalm. But the journey doesn't end there. The mood suddenly changes in verse four:

Even though I walk through the darkest valley,
I will fear no evil,
for you are with me;
your rod and your staff,
they comfort me.

David is now walking through "the darkest valley", which sounds grim. In other translations it is referred to as the "valley of the shadow of death" – which sounds like a very bad place indeed.

Yet all of us are there at one time or another in our lives. Many of us know what it is like to grieve for people close to us. Many of us know what it is like to have serious illness or mental health problems, to be separated from our families, to see an important relationship break down or to lose the job that we loved.

So much as Psalm 23 is about the good times, it's also about the bad times. David wrote psalms not as a theological textbook. He didn't sit down and try to tell us what God was like from things he'd learned in a classroom or been taught by a wise person. He wrote psalms about his relationship with God during the good times and the bad.

David had amazing highs but he also had crushing lows. He was hunted down by Saul whom he had once loved. He went on the run and it looked like he and his men were going to be wiped out. He fell out with various people and acted selfishly. He fell to such an extent that he had an affair with another man's wife and then had the man murdered to cover it up. You can't get much lower than that. Yet despite the trials, he was able to write in Psalm 23: "Yea, though I walk through the valley of the shadow of death, I will fear no evil: for thou art with me; thy rod and thy staff they comfort me". Even in the worst of it, God is there.

It's no coincidence that when he has come through the valley of the shadow of death on this journey, David's tone and the tense in which he writes changes slightly. He now talks less about his experience with God, but about his hope for the future:

You prepare a table before me
in the presence of my enemies.
You anoint my head with oil;
my cup overflows.
Surely your goodness and love will follow me
all the days of my life,
and I will dwell in the house of the Lord forever.

This final part could be read in a very plain way to say that David knows everything is going to be alright, because God is with him. But this is the point we realise the psalm is not just about David and God. It's about us – the people of God and our relationship to him.

It is an eschatological passage – one that looks forward to the end of time, when the great shepherd, Jesus, will invite us, His Church, to a banquet which lasts forever. Suddenly the shepherd and sheep imagery is gone. We're no longer in need of it because David is giving it to us straight. Through Jesus, we will be invited to God's banquet forever.

It's a psalm for good times, for bad times, of hope for the future. Yet it's only half as effective as it might be if it's just on the page. This is a psalm to be read and sung and recited to each other and to ourselves at all times. The things we repeat can come back to us when we need them. Psalm 23 should be second nature to us, so that in good times or bad, when looking to the past or the future we can bring it to mind.

Follow Andy Walton on Twitter @waltonandy