I like the church I belong to very much. I'm one of the leaders there but we have a full-time pastor who does the heavy lifting. I do quite a bit there, but not half as much as some people. I rarely get to the point where I think, "Not church again."

But I sometimes recall a conversation – let's call it that, though it was more like a row – I had when I was in pastoral ministry a few years ago. The gentleman in question wasn't from my church and I can't remember how the subject came up, but he was absolutely adamant that when pastors had a Sunday off they should still go to their church. It was their duty. As well as being pastors they were church members (we were Baptists, that's how it works) and that's what church members do.

No, I said. When I go to church and I know someone else is in charge, it's more stressful than taking the service myself. What happens if they get something wrong? What happens if their sermon is rubbish? Even worse, what happens if it's brilliant – ie, better than mine? And why is no one talking to that newcomer, and who's on the rota to wash the communion cups, and is it OK if I go home and leave someone else to lock up?

And I like visiting different churches and seeing how people do things there. And sometimes – not often, but sometimes – I might decide I don't want to go at all.

It's not just pastors who feel like that. Church is usually a place of blessing; it feeds us. But sometimes, because of the commitments we undertake when we're there or the demands of the people we meet there, it can be draining. It doesn't feel like rest, it feels like work.

And the worst of it is when we're expected to be there every week, rain or shine, tired or not.

I know all the arguments for going to church when you don't feel like it. I've used them myself, and by and large I believe them. It's too easy to let other things crowd out our service of God. The discipline of attendance is good. I have no patience at all with the "You don't have to go to church to be a Christian" excuse for staying in bed on Sunday mornings.

But I believe we need to be able to tell ourselves sometimes it's OK not to go, or to visit another church – and that pastors ought to be fine with that.

The Sabbath is a very rich, powerful idea. It says we aren't just machines, who produce things for ourselves or other people to consume. Rest is built in to the nature of things.

We sometimes get confused about the Sabbath. We think the Christian Sunday is the Jewish Sabbath, which it isn't. We think that the duty to keep the Sabbath translates to going to church on Sunday. It's quite hard to see where that idea came from, but even so we should remember that some of Jesus' most confrontational moments were about the Sabbath. The Pharisees thought it was about things you didn't do; Jesus knew it was about things you did. He says to them in Mark 2:27: "The Sabbath was made for man, not man for the Sabbath."

The Sabbath is about rest, not about more work.

Sometimes we need a Sabbath from Sundays.

I hope that if I were still in pastorate, I'd have the courage to preach that it's OK not to go to church. Pastors invest a lot in their congregations. They hope for investment back, and tend to judge that by how many people show up. It's not necessarily a good way of measuring spiritual health.

We need to feel that we aren't judged if we take a walk in the country or have some family time – particularly with the widespread Sunday working we have now, which means that fewer people have a day they can share with those they love most.

We need to feel that God can meet us outside the walls of the church as well as inside. And pastors need to feel confident enough in their people's commitment to Christ to be able to release them to find him where they need to, rather than where they think they ought to.

I believe in church and churchgoing, but I know God is bigger than Sundays. If I don't go to church, I'm no less near to him, or He to me.

Follow Rev Mark Woods on Twitter.