The thing about pair programming is that it’s similar to other closely-collaborative creative activities. If you perform pair programming with someone you wouldn’t marry or write a novel with, you’re going to have a bad time, because it requires the same kind of skills and chemistry.
Pair programming involves having somebody point out your stupidest, most embarassing mistakes and then acknowledging and fixing them. And then you switch roles. Most of us only have the kind of relationship that allows us to be so critical without permanent damage with one or two people.
(Now, having someone of a greater skill level behind you and not switching roles is not pair programming: it’s one on one instruction. That’s fine, but it’s different because the power dynamic is different.)