AMONG MY perfect days of the year, none counts higher than that first morning when, the car loaded, you set off on holiday. Nowadays, it’s always in the same direction — a similar routine, because we go to our house in the Tarn region of France. We’ve had every August holiday there since 1995.

I enjoy the preparation for a holiday. I like going out to buy a new T-shirt and deciding what books to take. I like getting my tennis racket restrung and buying new walking shoes, which, inevitably, I don’t break in, so I get blisters. If I’m driving across France, I normally know where there’s a church with a good romanesque painting in the nave, or whatever. My family has been very tolerant of my tendency to plan holidays as though they were a sort of Panzer campaign.

When Lavender and I were first married, we had three perfect holidays following what Freda White, in her classic travel book, called the Three Rivers of France: the Dordogne, the Lot and the Tarn. We looked for places in the Michelin guide that had food at reasonable prices and we stayed in a marvellous hotel, Le Belle Rive, in a village called Najac.

One of the walks we used to take goes within about 300 yards of the farmhouse we now own. It’s a glorious piece of la France profonde, where the rivers cut through the limestone, making spectacular gorges. The main enthusiasm I pursue is gardening. Next time we go, I’ll be passionately interested in whether or not the wonderful woman who helps with the garden thinks it’s the right time to plant a new olive tree.

I also play tennis, and we walk and read a lot. We have long lunches under the cherry tree and late suppers in the courtyard. As the sun sets, it produces a wonderful dappled-light effect on the wall of the house.

Of course, I like doing other things as well. We had a walking holiday in Italy at the end of May, with two friends, which was heaven. We walked to Urbino across the ridges of Le Marche — people who take Hadrian’s Wall in their stride would doubtless think we were pretty niminy-piminy, but we did the best part of 60 miles in five or six days, which made me realise I wasn’t fit enough.

Urbino itself is wonderful, especially the Ducal Palace, sited at the top of the city. It has some great paintings, including Piero della Francesca’s The Flagellation. We’re both great explorers of architecture ... on one holiday, when we had taken in three or four chateaux on the Loire, we got to Chenonceaux and the children simply sat in the back of the car and went on strike, so we had to do it rather quickly on our own.

We’ve never had a really grim holiday, though I can remember a difficult one my first summer after getting into the House of Commons. We were holidaying on the north Cornwall coast, at a hotel that had been widely recommended for its food. Trouble was, the owner had since changed. It was the time of the gales that sank so many boats in the Fastnet race. And we sat in four sweaters, shivering on the beach, with our dog getting covered in oil, worrying about how we were going to find another five grand to buy our flat in Westminster.

When I first went into the cabinet in 1989, we did a house swap with a family in Berkeley, California, then drove down to stay just outside Hollywood with a friend who’s a film producer. The children all adored the sense of being close to celebrity. But don’t ask me to tell you who they met; my children say the definition of a celebrity is somebody I’ve never heard of.

We had some good holidays from Hong Kong, including one in the Philippines at Chinese new year. It was at the Amanpulo, and it’s the only time I’ve stayed in the sort of luxury to which my children would have loved to become accustomed. It was just silver sandy beaches, exotic tropical foliage, little bungalows set among the trees with infinity swimming pools and the sound of the water lapping on the sand. I’d be far too claustrophobic to dive, but I like lying on the surface, looking at the fish below.

Times recommends

• The 20 best winter sun destinations
• Australia's five best roadtrips
• The Scotland shooting lodge that targets greenies