Modern Life: Conversations with my Window Cleaner

'Where are you off to,anywhere nice ?'
'The Isle of Wight,'
'Oh lovely,used to go there as a kid'
'Never been there myself,my mum lives there though,'
'Oh it's very nice, and I'll tell you something, you don't get any black people there.'

Now I don't know my window cleaner very well, and after the above conversation,less so than I thought. I'm still getting used to the idea that I even have a window cleaner, but well the windows were dirty and he was in the road and the wife said, 'the windows are very dirty.' So that was that.
I've always thought he was a nice chap, he'd sometimes stop for a cup of tea and ask if I followed the football, which I didn't and occasionally he'd leave his ladders around the back because he didn't have a car.
So I'm a bit surprised to discover I've got a racist window cleaner.
Is he a racist ? Or was he making some sort of joke about black and white, a play on words with the Isle of Wight ?
Maybe he thinks I'm a racist ?
Do I look like one ?
Is that racist ?
What about me has said that's ok ? Why does he think I might choose my holiday destination on the basis of skin colour ? Maybe he's peered through the window whilst cleaning it and seen my copy of Bridget Hitler's Diary and thinks it won't be an issue.
What's more worrying is I'm starting to sound like David Mitchell, Mark from Peep Show. The thing is I just don't know him well enough to ask what he meant, we're not exactly drinking buddies, which after another revelation, it turns out, is just as well:
'I'll get my car back in a month,'
'Oh, what happened?'
'Got banned didn't I. Drink driving. They test you though,now,have to be clean for a month. They test your hair. I'm doing it now. Tough going.'
Is it wise for him to be near all that window cleaner, I mean,the temptation.

I scuttle back indoors. The wife returns and I tell her, she's shocked.
'You didn't tell him we were going on holiday did you ?'
I shrug. I'm resigned to it now. Yesterday he just cleaned my windows every month. Now I fully expect to return to find our house broken into whilst someone runs a BNP rally/AA meeting in the front room. Better hide my copy of Mein Kampf though, it is signed.