Don't look at me, I've never lost a chance to try to lure people to be interested in joining when I thought you would approve of them. Not my fault no one has accepted yet, not everyone can get away with dragging people around for their own best like a mother would.

[He just seems to have bad luck at recruiting, or maybe it's because he prefers it if they go to Haruhi rather than him having to go and be the meeting link. With all the effort she puts in advertising and recruiting he just feels it would be better if more people got into the village that way. Would also avoid conflict of 'he said you're fine but I don't like you' or whatever.]